View mode: basic / threaded / horizontal-split · Log in · Help
March 22, 2011
Strategies for resolving cyclic dependencies in static ctors
I'm intending this thread as somewhat of a roundtable-like discussion. 
Hopefully we can come up with good material for a short article on Wiki4D, 
or maybe the D website, or wherever.

The scenario: A coder is writing some D, compiles, runs and gets a "Cyclic 
dependency in static ctors" error. Crap! A pain for experienced D users, and 
very bad PR for new D users. (Hopefully we'll eventually get some sort of 
solution for this, but in the meantime D users just have to deal with it.)

The question: What now? What strategies do people find useful for dealing 
with this? Any specific "first steps" to take? Best practices? Etc.

Aside from the old "start merging modules" PITA that many of us are familiar 
with from the old "100 forward reference errors at the drop of a hat" days, 
I've found one viable (but still kinda PITA) strategy so far:

1. Look at the line that says: "module foo -> module bar -> module foo"

2. Pick one of those two modules (whichever has the simplest static ctors) 
and eliminate all the static ctors using the following pattern:

The pattern: The trick is to convert every variable that needs to be 
initialized into an "init on first use" ref @property. This "ref @property" 
checks a "hasThisBeenInited" bool and, if false, runs all the 
initialization. If the variable is a reference type, then *sometimes* you 
can get away with just checking for null (though often not, because it will 
just get-reinited ).

Example:
------------------------------
// Old:
class Foo { /+...+/ }
int number;
Foo foo;
static this
{
   foo = new Foo();
   number = /+ something maybe involving foo +/;
}

// New:
class Foo { /+...+/ }

private int _number;
@property ref int number()
{
   forceModuleInit();
   return _number;
}

private Foo _foo;
@property ref Foo foo()
{
   forceModuleInit();
   return _foo;
}

bool isModuleInited=false;
static void forceModuleInit() // Hopefully inlined
{
   if(!isModuleInited)
   {
       staticThis();
       isModuleInited = true;
   }
}
static void staticThis() // Workaround for static ctor cycle
{
   foo = new Foo();
   number = /+ something maybe involving foo +/;
}
------------------------------

If one of the variables being inited in the static ctor is something you 
*know* will never be accessed before some other specific variable is 
accessed, then you can skip converting it to "@property ref" if you want.

It's a big mess, but the conversion can be done deterministically, and even 
mechanically (heck, a ctfe string mixin wrapper could probably be built to 
do it).

The potential downsides:

1. If you come across an @propery bug or limitation, you're SOL. This should 
become less and less of an issue with time, though.

2. If one of the variables you converted is frequently-accessed, it could 
cause a performance problem.

3. Small increase to storage requirements. Might potentially be a problem if 
it's within templated or mixed-in code that gets instantiated many times.

At one point, I fiddled around with the idea of converting static ctors to 
"staticThis()" and then having one real static ctor for the entire library 
(assuming it's a library) that manually calls all the staticThis functions. 
One problem with this is that it's easy to accidentally forget to call one 
of the staticThis functions. The other big problem I found with this though, 
especially for a library, is that it requires everyone importing your code 
to always import through a single "import foo.all" module. If some user 
skips that, then the static ctors won't get run. There might be some 
possible workarounds for that, though:

- If the library has some primary interface that always gets used, then that 
can easily check if the static ctors have run and error out if not. If the 
primary interface is *always* the first part of your library used (or at 
least the first of all the parts that actually rely on the static ctors 
having run), then you could even run the static ctors right then instead of 
erroring out. That's a lot of "if"s, though, so it may not be 
widely-applicable.

- If you convert *all* static ctors to staticThis(), it might be possible to 
stick the one main static ctor into a private utility module that gets 
privately imported by all modules in the library. Then users can continue 
importing whatever module(s) they want. But if you don't convert *all* of 
the static ctors to staticThis, then you'll just re-introduce a cycle.

But if there's ever two separate libraries that have any interdependencies, 
then the one-main-real static ctor (that calls all the staticThis() funcs) 
will have to be shared between the two libraries. So overall, this approach 
may be possible, but maybe only in certain cases, and can involve a lot of 
changes.
March 22, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
"Nick Sabalausky" <a@a.a> wrote in message 
news:im8pmp$18p7$1@digitalmars.com...
>
> The pattern: The trick is to convert every variable that needs to be 
> initialized into an "init on first use" ref @property. This "ref 
> @property" checks a "hasThisBeenInited" bool and, if false, runs all the 
> initialization. If the variable is a reference type, then *sometimes* you 
> can get away with just checking for null (though often not, because it 
> will just get-reinited ).
>

...if anyone sets it to null.

(Forgot to finish that last sentence.)
March 22, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On Tue, 22 Mar 2011 02:12:55 +0200, Nick Sabalausky <a@a.a> wrote:

> The question: What now? What strategies do people find useful for dealing
> with this? Any specific "first steps" to take? Best practices? Etc.

Your post doesn't seem to mention it, but how about converting the static  
ctors to initialization functions, and calling them from a single static  
ctor within the dependency loop?

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net
March 22, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On Tue, 22 Mar 2011 04:30:37 +0200, Vladimir Panteleev  
<vladimir@thecybershadow.net> wrote:

> Your post doesn't seem to mention it,

Sorry, didn't scroll down enough :)

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net
March 22, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
"Vladimir Panteleev" <vladimir@thecybershadow.net> wrote in message 
news:op.vsp3zooituzx1w@cybershadow.mshome.net...
> On Tue, 22 Mar 2011 04:30:37 +0200, Vladimir Panteleev 
> <vladimir@thecybershadow.net> wrote:
>
>> Your post doesn't seem to mention it,
>
> Sorry, didn't scroll down enough :)
>

Well, that is a lot of scrolling, actually :)
March 22, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On 03/22/2011 01:12 AM, Nick Sabalausky wrote:
> I'm intending this thread as somewhat of a roundtable-like discussion.
> Hopefully we can come up with good material for a short article on Wiki4D,
> or maybe the D website, or wherever.
>
> The scenario: A coder is writing some D, compiles, runs and gets a "Cyclic
> dependency in static ctors" error. Crap! A pain for experienced D users, and
> very bad PR for new D users. (Hopefully we'll eventually get some sort of
> solution for this, but in the meantime D users just have to deal with it.)
>
> The question: What now? What strategies do people find useful for dealing
> with this? Any specific "first steps" to take? Best practices? Etc.
>
> Aside from the old "start merging modules" PITA that many of us are familiar
> with from the old "100 forward reference errors at the drop of a hat" days,
> I've found one viable (but still kinda PITA) strategy so far:
>
> 1. Look at the line that says: "module foo ->  module bar ->  module foo"
>
> 2. Pick one of those two modules (whichever has the simplest static ctors)
> and eliminate all the static ctors using the following pattern:
>
> The pattern: The trick is to convert every variable that needs to be
> initialized into an "init on first use" ref @property. This "ref @property"
> checks a "hasThisBeenInited" bool and, if false, runs all the
> initialization. If the variable is a reference type, then *sometimes* you
> can get away with just checking for null (though often not, because it will
> just get-reinited ).
>
> Example:
> ------------------------------
> // Old:
> class Foo { /+...+/ }
> int number;
> Foo foo;
> static this
> {
>      foo = new Foo();
>      number = /+ something maybe involving foo +/;
> }
>
> // New:
> class Foo { /+...+/ }
>
> private int _number;
> @property ref int number()
> {
>      forceModuleInit();
>      return _number;
> }
>
> private Foo _foo;
> @property ref Foo foo()
> {
>      forceModuleInit();
>      return _foo;
> }
>
> bool isModuleInited=false;
> static void forceModuleInit() // Hopefully inlined
> {
>      if(!isModuleInited)
>      {
>          staticThis();
>          isModuleInited = true;
>      }
> }
> static void staticThis() // Workaround for static ctor cycle
> {
>      foo = new Foo();
>      number = /+ something maybe involving foo +/;
> }
> ------------------------------
>
> If one of the variables being inited in the static ctor is something you
> *know* will never be accessed before some other specific variable is
> accessed, then you can skip converting it to "@property ref" if you want.
>
> It's a big mess, but the conversion can be done deterministically, and even
> mechanically (heck, a ctfe string mixin wrapper could probably be built to
> do it).
>
> The potential downsides:
>
> 1. If you come across an @propery bug or limitation, you're SOL. This should
> become less and less of an issue with time, though.
>
> 2. If one of the variables you converted is frequently-accessed, it could
> cause a performance problem.
>
> 3. Small increase to storage requirements. Might potentially be a problem if
> it's within templated or mixed-in code that gets instantiated many times.
>
> At one point, I fiddled around with the idea of converting static ctors to
> "staticThis()" and then having one real static ctor for the entire library
> (assuming it's a library) that manually calls all the staticThis functions.
> One problem with this is that it's easy to accidentally forget to call one
> of the staticThis functions. The other big problem I found with this though,
> especially for a library, is that it requires everyone importing your code
> to always import through a single "import foo.all" module. If some user
> skips that, then the static ctors won't get run. There might be some
> possible workarounds for that, though:
>
> - If the library has some primary interface that always gets used, then that
> can easily check if the static ctors have run and error out if not. If the
> primary interface is *always* the first part of your library used (or at
> least the first of all the parts that actually rely on the static ctors
> having run), then you could even run the static ctors right then instead of
> erroring out. That's a lot of "if"s, though, so it may not be
> widely-applicable.
>
> - If you convert *all* static ctors to staticThis(), it might be possible to
> stick the one main static ctor into a private utility module that gets
> privately imported by all modules in the library. Then users can continue
> importing whatever module(s) they want. But if you don't convert *all* of
> the static ctors to staticThis, then you'll just re-introduce a cycle.
>
> But if there's ever two separate libraries that have any interdependencies,
> then the one-main-real static ctor (that calls all the staticThis() funcs)
> will have to be shared between the two libraries. So overall, this approach
> may be possible, but maybe only in certain cases, and can involve a lot of
> changes.

I think the idea of a single static constructor in a main lib import module 
(let's call it lib.d), calling staticThis func in every module, is the right 
track. First, it is rather a good practice (I mean both from the designer and 
user points of view).
Upon "it's easy to accidentally forget to call one of the staticThis 
functions": just systematically write one in every module, possibly empty at 
start. Then, from the main module, call staticThis on every imported module. 
This can be a cascade: staticThis if M calls staticThis of its own private 
dependencies. Shared submodules should still be init from lib, but even double 
init would be cheap thank the the bool flag.

Upon the case where users may need and import only part of your lib, then I 
guess obviously this mean some kind of *independency*, doesn't it? Else, all 
must be init-ed, and thus imported, anyway, don't you think? In which case they 
could as well import the main module in every case, and use only what they need.

Upon the last issue of common dependancies. If the problem of cyclic 
dependencies in static ctors is analog to circular imports, then there are 2 
common strategies (here considering 2 modules importing each other):
* Isolate a part of a module that requires its own module and another one, 
place it in a 3rd module which imports both. (This is a common issue for test 
cases.)
* Conversely, isolate a part of a module that /is/ required by its own module 
and another one, place it in a 3rd module imported by both. (This is a common 
issue for tool features.)
Strangely enough, this often solves the problem by splitting further instead of 
by merging. Though I don't know whether this is appropriate for your issue.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com
March 22, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On 03/22/2011 02:12 AM, Nick Sabalausky wrote:
> I'm intending this thread as somewhat of a roundtable-like discussion.
> Hopefully we can come up with good material for a short article on Wiki4D,
> or maybe the D website, or wherever.
>
> The scenario: A coder is writing some D, compiles, runs and gets a "Cyclic
> dependency in static ctors" error. Crap! A pain for experienced D users, and
> very bad PR for new D users. (Hopefully we'll eventually get some sort of
> solution for this, but in the meantime D users just have to deal with it.)
>
> The question: What now? What strategies do people find useful for dealing
> with this? Any specific "first steps" to take? Best practices? Etc.

One commonly used hack is to move static constructors into a separate 
helper module and call the initialization function via a C extern (like 
it is done in std.stdiobase):

----
module foo_helper;

private extern(C) foo_static_ctor();
static this()
{
    foo_static_ctor();
}

-----

module foo;
import foo_helper;

private Object global;
private extern(C) void foo_static_ctor()
{
    global = new Object;
}
----

Note that "global" is guaranteed to have been initialized when accessed 
from static constructors in modules that import "a". Being able to 
instruct the compiler to do this implicitly (so we could put static 
ctors in templates, for example) would probably solve most of static 
ctor problems.

>
> Aside from the old "start merging modules" PITA that many of us are familiar
> with from the old "100 forward reference errors at the drop of a hat" days,
> I've found one viable (but still kinda PITA) strategy so far:
>
> 1. Look at the line that says: "module foo ->  module bar ->  module foo"
>
> 2. Pick one of those two modules (whichever has the simplest static ctors)
> and eliminate all the static ctors using the following pattern:
>
> The pattern: The trick is to convert every variable that needs to be
> initialized into an "init on first use" ref @property. This "ref @property"
> checks a "hasThisBeenInited" bool and, if false, runs all the
> initialization. If the variable is a reference type, then *sometimes* you
> can get away with just checking for null (though often not, because it will
> just get-reinited ).
>
> Example:
> ------------------------------
> // Old:
> class Foo { /+...+/ }
> int number;
> Foo foo;
> static this
> {
>      foo = new Foo();
>      number = /+ something maybe involving foo +/;
> }
>
> // New:
> class Foo { /+...+/ }
>
> private int _number;
> @property ref int number()
> {
>      forceModuleInit();
>      return _number;
> }
>
> private Foo _foo;
> @property ref Foo foo()
> {
>      forceModuleInit();
>      return _foo;
> }
>
> bool isModuleInited=false;
> static void forceModuleInit() // Hopefully inlined
> {
>      if(!isModuleInited)
>      {
>          staticThis();
>          isModuleInited = true;
>      }
> }
> static void staticThis() // Workaround for static ctor cycle
> {
>      foo = new Foo();
>      number = /+ something maybe involving foo +/;
> }
> ------------------------------
>
> If one of the variables being inited in the static ctor is something you
> *know* will never be accessed before some other specific variable is
> accessed, then you can skip converting it to "@property ref" if you want.
>
> It's a big mess, but the conversion can be done deterministically, and even
> mechanically (heck, a ctfe string mixin wrapper could probably be built to
> do it).
>
> The potential downsides:
>
> 1. If you come across an @propery bug or limitation, you're SOL. This should
> become less and less of an issue with time, though.
>
> 2. If one of the variables you converted is frequently-accessed, it could
> cause a performance problem.
>
> 3. Small increase to storage requirements. Might potentially be a problem if
> it's within templated or mixed-in code that gets instantiated many times.

4. Initializing shared data needs synchronization. Then, your example 
would look similar to this:

------------------------------

class Foo { /+...+/ }

private immutable int _number;
@property ref int number()
{
     forceModuleInit();
     return _number;
}

private immutable Foo _foo;
@property immutable(Foo) foo()
{
     forceModuleInit();
     return _foo;
}

bool isModuleInited=false;
__gshared bool isSharedModuleInited=false;
static void forceModuleInit() // Hopefully inlined
{
     if(!isModuleInited)
     {
         synchronized(someLock)
         {
             if (!isSharedModuleInited)
             {
                 staticThis();
                 isSharedModuleInited = true;
             }
         }
         isModuleInited = true;
     }
}
static void sharedStaticThis() // Workaround for shared static ctor cycle
{
     auto foo = new Foo();
     _number = /+ something maybe involving foo +/;
     _foo = cast(immutable)foo;
}
------------------------------

>
> At one point, I fiddled around with the idea of converting static ctors to
> "staticThis()" and then having one real static ctor for the entire library
> (assuming it's a library) that manually calls all the staticThis functions.
> One problem with this is that it's easy to accidentally forget to call one
> of the staticThis functions. The other big problem I found with this though,
> especially for a library, is that it requires everyone importing your code
> to always import through a single "import foo.all" module. If some user
> skips that, then the static ctors won't get run. There might be some
> possible workarounds for that, though:
>
> - If the library has some primary interface that always gets used, then that
> can easily check if the static ctors have run and error out if not. If the
> primary interface is *always* the first part of your library used (or at
> least the first of all the parts that actually rely on the static ctors
> having run), then you could even run the static ctors right then instead of
> erroring out. That's a lot of "if"s, though, so it may not be
> widely-applicable.
>
> - If you convert *all* static ctors to staticThis(), it might be possible to
> stick the one main static ctor into a private utility module that gets
> privately imported by all modules in the library. Then users can continue
> importing whatever module(s) they want. But if you don't convert *all* of
> the static ctors to staticThis, then you'll just re-introduce a cycle.
>
> But if there's ever two separate libraries that have any interdependencies,
> then the one-main-real static ctor (that calls all the staticThis() funcs)
> will have to be shared between the two libraries. So overall, this approach
> may be possible, but maybe only in certain cases, and can involve a lot of
> changes.
>
>
>
March 22, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On 2011-03-22 01:12, Nick Sabalausky wrote:
> At one point, I fiddled around with the idea of converting static ctors to
> "staticThis()" and then having one real static ctor for the entire library
> (assuming it's a library) that manually calls all the staticThis functions.
> One problem with this is that it's easy to accidentally forget to call one
> of the staticThis functions. The other big problem I found with this though,
> especially for a library, is that it requires everyone importing your code
> to always import through a single "import foo.all" module. If some user
> skips that, then the static ctors won't get run. There might be some
> possible workarounds for that, though:

One idea could be, although very platform dependent, to iterate the 
symbol table, finding all "staticThis" functions and calling them. Then 
you don't have the problem of forgetting to call one of those functions.

-- 
/Jacob Carlborg
March 22, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
> On 03/22/2011 02:12 AM, Nick Sabalausky wrote:
> > I'm intending this thread as somewhat of a roundtable-like discussion.
> > Hopefully we can come up with good material for a short article on
> > Wiki4D, or maybe the D website, or wherever.
> > 
> > The scenario: A coder is writing some D, compiles, runs and gets a
> > "Cyclic dependency in static ctors" error. Crap! A pain for experienced
> > D users, and very bad PR for new D users. (Hopefully we'll eventually
> > get some sort of solution for this, but in the meantime D users just
> > have to deal with it.)
> > 
> > The question: What now? What strategies do people find useful for dealing
> > with this? Any specific "first steps" to take? Best practices? Etc.
> 
> One commonly used hack is to move static constructors into a separate
> helper module and call the initialization function via a C extern (like
> it is done in std.stdiobase):

That's what Phobos does to solve the problem (std.stdiobase being only one of 
the places that it does it). It's likely the solution that I would use as 
well.

- Jonathan M Davis
March 22, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On 2011-03-22 05:16:31 -0400, Max Samukha <max@spam.box> said:

> ----
> module foo_helper;
> 
> private extern(C) foo_static_ctor();
> static this()
> {
>      foo_static_ctor();
> }
> 
> -----
> 
> module foo;
> import foo_helper;
> 
> private Object global;
> private extern(C) void foo_static_ctor()
> {
>      global = new Object;
> }
> ----
> 
> Note that "global" is guaranteed to have been initialized when accessed 
> from static constructors in modules that import "a".

I don't know why people keep repeating that falacy. This statement is 
true only as long as there are no circular dependencies. It should 
read: "'global' is guarentied to have been initialized when access from 
static constructors in module that import 'a' _and_ which are not 
imported by 'a', directly or indirectly."

Because once you introduce a circular dependency, you get this:

----
module foo_helper;

private extern(C) foo_static_ctor();
static this() { foo_static_ctor(); }
-----
module foo;
import foo_helper;
import bar;

private Object global;
private extern(C) void foo_static_ctor()
{
    global = new Object;
    bar.testGlobal();
}
public void testGlobal()
{
    assert(global);
}
----
module bar_helper;

private extern(C) bar_static_ctor();
static this() { bar_static_ctor(); }
-----
module bar;
import bar_helper;
import foo;

private Object global;
private extern(C) void bar_static_ctor()
{
    global = new Object;
    foo.testGlobal();
}
public void testGlobal()
{
    assert(global);
}
----

Note how foo_static_ctor() and bar_static_ctor() each calls a function 
that needs the global variable of the other module to be initialized. 
It should be obvious that it can't work. If you doubt me, try it.


> Being able to instruct the compiler to do this implicitly (so we could 
> put static ctors in templates, for example) would probably solve most 
> of static ctor problems.

It'd have the exact same effect as adding a pragma to bypass the check 
for circular dependencies, while making things more complicated.


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/
« First   ‹ Prev
1 2 3
Top | Discussion index | About this forum | D home