View mode: basic / threaded / horizontal-split · Log in · Help
September 19, 2012
static init cycle detection problem
I am struggeling to get around the cycle detection kicking in 
when I have static init in modules that depend on eachother.

I have seen some threads on 'fixes' for this, e.g. adding a 
@standalone property to the module or similar. Has there been any 
progress on this?

If not would it be possible in e.g. main() to get a list of all 
compiled-in modules, and then iterate over them and call an init 
function where it exists? As long as there is a way to list the 
name of the modules at compile-time, this should be pretty easy..?

-Øivind
September 19, 2012
Re: static init cycle detection problem
On Wed, 19 Sep 2012 22:25:46 +0200, Øivind <oivind.loe@gmail.com> wrote:

> I am struggeling to get around the cycle detection kicking in when I  
> have static init in modules that depend on eachother.
>
> I have seen some threads on 'fixes' for this, e.g. adding a @standalone  
> property to the module or similar. Has there been any progress on this?
>
> If not would it be possible in e.g. main() to get a list of all  
> compiled-in modules, and then iterate over them and call an init  
> function where it exists? As long as there is a way to list the name of  
> the modules at compile-time, this should be pretty easy..?

There's no way to get that list at compile-time, because object files may
be added at link-time. However, D has a ModuleInfo object, which contains
information on all modules in the program:

import std.stdio;
void main( ) {
    foreach( m; ModuleInfo ) {
        writeln( m.name );
    }
}

For details on how this object works, have a look-see at
src/druntime/src/object_.d in your DMD installation folder.

I'm not sure what you're asking for is possible even given this object,
but it's probably the closest you'll (easily) get.

-- 
Simen
September 19, 2012
Re: static init cycle detection problem
On Wednesday, 19 September 2012 at 20:56:17 UTC, Simen Kjaeraas 
wrote:
> On Wed, 19 Sep 2012 22:25:46 +0200, Øivind 
> <oivind.loe@gmail.com> wrote:
>
>> I am struggeling to get around the cycle detection kicking in 
>> when I have static init in modules that depend on eachother.
>>
>> I have seen some threads on 'fixes' for this, e.g. adding a 
>> @standalone property to the module or similar. Has there been 
>> any progress on this?
>>
>> If not would it be possible in e.g. main() to get a list of 
>> all compiled-in modules, and then iterate over them and call 
>> an init function where it exists? As long as there is a way to 
>> list the name of the modules at compile-time, this should be 
>> pretty easy..?
>
> There's no way to get that list at compile-time, because object 
> files may
> be added at link-time. However, D has a ModuleInfo object, 
> which contains
> information on all modules in the program:
>
> import std.stdio;
> void main( ) {
>     foreach( m; ModuleInfo ) {
>         writeln( m.name );
>     }
> }
>
> For details on how this object works, have a look-see at
> src/druntime/src/object_.d in your DMD installation folder.
>
> I'm not sure what you're asking for is possible even given this 
> object,
> but it's probably the closest you'll (easily) get.

If I define a function:

void getMembers() {

}

in one of my modules, all modules including this get the 
xgetMembers property of the ModuleInfo struct set. It seems like 
I should be able to do something with this, but i then get the 
following assertion failure:

./dboss(_d_assertm+0x26) [0x8aa1da]
./dboss() [0x7c8555]
./dboss(boss.core.proc.ProcBase!(boss.core.boss.Boss, 
"boss").ProcBase boss.core.proc.ProcBase!(boss.core.boss.Boss, 
"boss").ProcBase.__ctor()+0x31) [0x797805]
./dboss(boss.core.boss.Boss 
boss.core.boss.Boss.__ctor(immutable(char)[][])+0x21) [0x791099]
./dboss(_Dmain+0x33) [0x7488fb]
./dboss(extern (C) int rt.dmain2.main(int, char**).void 
runMain()+0x1c) [0x8aaa70]
./dboss(extern (C) int rt.dmain2.main(int, char**).void 
tryExec(scope void delegate())+0x2a) [0x8aa3ea]
./dboss(extern (C) int rt.dmain2.main(int, char**).void 
runAll()+0x3b) [0x8aaab7]
./dboss(extern (C) int rt.dmain2.main(int, char**).void 
tryExec(scope void delegate())+0x2a) [0x8aa3ea]
./dboss(main+0xd1) [0x8aa375]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) 
[0x7f1d0edcc76d]

Removing the function from the module makes the assertion failure 
go away.. What is the cause of this? And what is the getMembers 
function used for?


Another way of approaching this would be if I could feed a list 
of modules into DMD during compile time. In C++, i would be able 
to do this by passing a define to g++ on the command line when 
invoking it. Is it possible to do something similar with DMD? 
E.g. create a list of modules before invoking DMD, pass it to 
DMD, and use this at compile time to import and call my init 
function?
September 19, 2012
Re: static init cycle detection problem
> Another way of approaching this would be if I could feed a list 
> of modules into DMD during compile time. In C++, i would be 
> able to do this by passing a define to g++ on the command line 
> when invoking it. Is it possible to do something similar with 
> DMD? E.g. create a list of modules before invoking DMD, pass it 
> to DMD, and use this at compile time to import and call my init 
> function?

I can of course just generate a separate init.d file containing 
all of this info and then use that for static init of my modules. 
Think that should work :)
September 20, 2012
Re: static init cycle detection problem
On Wednesday, September 19, 2012 22:25:46 Øivind wrote:
> I am struggeling to get around the cycle detection kicking in
> when I have static init in modules that depend on eachother.
> 
> I have seen some threads on 'fixes' for this, e.g. adding a
> @standalone property to the module or similar. Has there been any
> progress on this?

The solution is to do something like what std.stdio does. Instead of having a 
static constructor, it has an extern(C) function which does the same, and then 
it has std/stdiobase.d which has a static constructor which calls that 
extern(C) function in std.stdio. As long as you don't actually have any inter-
dependencies, you can make it work.

Walter has refused suggestions which involve telling the compiler that there 
isn't actually dependency (via an attribute or whatnot) on the grounds that 
you're throwing away the compiler's checks, and it's too easy to screw up. In 
general, you can refactor things so that you don't have the circular 
dependency anymore, and if you can't, and the static constructors aren't 
actually forming a circular dependency, then you can pull the same trick that 
std.stdio does.

> If not would it be possible in e.g. main() to get a list of all
> compiled-in modules, and then iterate over them and call an init
> function where it exists? As long as there is a way to list the
> name of the modules at compile-time, this should be pretty easy..?

It will never be possible to determine the full list of modules at compile 
time due to C's linking model. For instance, you could link together modules 
which were built years apart. There's no way that they could know about each 
other. And since you could be using .di files, and the circular dependencies 
could actually be really indirect, the compiler can't possibly have all of the 
information that it needs to determine circular dependencies. A linker could 
do it but not as long as we use standard, C linkers. And that's not going to 
change.

So, the situation sucks, but there _are_ workarounds. The main one of course 
is to simply limit how much your modules import each other so that the few 
places that you need static constructors don't import each other even 
indirectly, but regardless, it can be worked around as long there isn't a true
circular dependency.

- Jonathan M Davis
September 20, 2012
Re: static init cycle detection problem
On Thursday, 20 September 2012 at 00:23:33 UTC, Jonathan M Davis 
wrote:
> On Wednesday, September 19, 2012 22:25:46 Øivind wrote:
>> I am struggeling to get around the cycle detection kicking in
>> when I have static init in modules that depend on eachother.
>> 
>> I have seen some threads on 'fixes' for this, e.g. adding a
>> @standalone property to the module or similar. Has there been 
>> any
>> progress on this?
>
> The solution is to do something like what std.stdio does. 
> Instead of having a
> static constructor, it has an extern(C) function which does the 
> same, and then
> it has std/stdiobase.d which has a static constructor which 
> calls that
> extern(C) function in std.stdio. As long as you don't actually 
> have any inter-
> dependencies, you can make it work.
>
> Walter has refused suggestions which involve telling the 
> compiler that there
> isn't actually dependency (via an attribute or whatnot) on the 
> grounds that
> you're throwing away the compiler's checks, and it's too easy 
> to screw up. In
> general, you can refactor things so that you don't have the 
> circular
> dependency anymore, and if you can't, and the static 
> constructors aren't
> actually forming a circular dependency, then you can pull the 
> same trick that
> std.stdio does.
>
>> If not would it be possible in e.g. main() to get a list of all
>> compiled-in modules, and then iterate over them and call an 
>> init
>> function where it exists? As long as there is a way to list the
>> name of the modules at compile-time, this should be pretty 
>> easy..?
>
> It will never be possible to determine the full list of modules 
> at compile
> time due to C's linking model. For instance, you could link 
> together modules
> which were built years apart. There's no way that they could 
> know about each
> other. And since you could be using .di files, and the circular 
> dependencies
> could actually be really indirect, the compiler can't possibly 
> have all of the
> information that it needs to determine circular dependencies. A 
> linker could
> do it but not as long as we use standard, C linkers. And that's 
> not going to
> change.
>
> So, the situation sucks, but there _are_ workarounds. The main 
> one of course
> is to simply limit how much your modules import each other so 
> that the few
> places that you need static constructors don't import each 
> other even
> indirectly, but regardless, it can be worked around as long 
> there isn't a true
> circular dependency.
>
> - Jonathan M Davis

Thanks for the explination. The trick you talk about has worked 
for me before, but now I really need static init of modules that 
depend on eachother. Only solution I can think of is to list all 
modules in a script and generate d code from this to call init 
functions.. Then pass it to dmd along with the other code.

Sad that we can't get this to work like it really should. D is a 
very flexible language in many ways, and it seems strange that it 
should not allow the stuff I want to do here..

-Øivind
September 20, 2012
Re: static init cycle detection problem
On Thursday, September 20, 2012 20:49:32 Øivind wrote:
> Thanks for the explination. The trick you talk about has worked
> for me before, but now I really need static init of modules that
> depend on eachother. Only solution I can think of is to list all
> modules in a script and generate d code from this to call init
> functions.. Then pass it to dmd along with the other code.
> 
> Sad that we can't get this to work like it really should. D is a
> very flexible language in many ways, and it seems strange that it
> should not allow the stuff I want to do here..

If you have a true circular dependency, then you're just plain screwed and 
need to redesign your code. That's fundamental and _can't_ be fixed.

Off the top of my head, the only reason that you wouldn't be able to use the 
trick that std.stdio uses but not have a true circular dependency is if your 
initializing const or immutable variables in a static constructor. But then 
the solution is to refactor your code so that you don't have the circular 
dependency, even if it's just moving those variables into a separate module, 
which is generally quite feasible, even if it's not necessarily quite what you 
wanted.

- Jonathan M Davis
September 20, 2012
Re: static init cycle detection problem
On Thu, 20 Sep 2012, Jonathan M Davis wrote:

> On Thursday, September 20, 2012 20:49:32 ?ivind wrote:
> > Thanks for the explination. The trick you talk about has worked
> > for me before, but now I really need static init of modules that
> > depend on eachother. Only solution I can think of is to list all
> > modules in a script and generate d code from this to call init
> > functions.. Then pass it to dmd along with the other code.
> > 
> > Sad that we can't get this to work like it really should. D is a
> > very flexible language in many ways, and it seems strange that it
> > should not allow the stuff I want to do here..
> 
> If you have a true circular dependency, then you're just plain screwed and 
> need to redesign your code. That's fundamental and _can't_ be fixed.
> 
> Off the top of my head, the only reason that you wouldn't be able to use the 
> trick that std.stdio uses but not have a true circular dependency is if your 
> initializing const or immutable variables in a static constructor. But then 
> the solution is to refactor your code so that you don't have the circular 
> dependency, even if it's just moving those variables into a separate module, 
> which is generally quite feasible, even if it's not necessarily quite what you 
> wanted.
> 
> - Jonathan M Davis
> 

I'm not picking on Jonathan's response, just needed an entry point to the 
thread...

Static initializers are overused in D, particularly phobos and druntime, 
imho.  They're a particular problem when it comes to use in library code.  
Mixing static and dynamic linkage along with initialization == things 
break in wierd ways.  I'd _much_ rather see them moved into explicit api 
calls or removed entirely where possible.

Every static initializtion is unavoidable (by users of the 
library) overhead.

My 2 cents,
Brad
September 24, 2012
Re: static init cycle detection problem
On Wed, 19 Sep 2012 16:25:46 -0400, Øivind <oivind.loe@gmail.com> wrote:

> I am struggeling to get around the cycle detection kicking in when I  
> have static init in modules that depend on eachother.
>
> I have seen some threads on 'fixes' for this, e.g. adding a @standalone  
> property to the module or similar. Has there been any progress on this?

No.  If two initializations truly don't depend on each other, you can  
initialize in their own module.

The issue is that you have to move *all* the initializations into their  
own module.  The compiler doesn't keep track of complex dependencies, it's  
all on a module-level.

I redesigned the static cycle detection code a while back (it was broken  
(allowed certain cycles) and much less informative), and I can say it was  
a very difficult problem to solve.

> If not would it be possible in e.g. main() to get a list of all  
> compiled-in modules, and then iterate over them and call an init  
> function where it exists? As long as there is a way to list the name of  
> the modules at compile-time, this should be pretty easy..?

No, you have no idea at compile-time what other modules will be linked in.

You need to do the cycle detection at link-time or run-time, and as long  
as we use standard C linkers, we cannot add that feature.

-Steve
Top | Discussion index | About this forum | D home