I'm intending this thread as somewhat of a roundtable-like discussion. Hopefully we can come up with good material for a short article on Wiki4D, or maybe the D website, or wherever.

The scenario: A coder is writing some D, compiles, runs and gets a "Cyclic dependency in static ctors" error. Crap! A pain for experienced D users, and very bad PR for new D users. (Hopefully we'll eventually get some sort of solution for this, but in the meantime D users just have to deal with it.)

The question: What now? What strategies do people find useful for dealing with this? Any specific "first steps" to take? Best practices? Etc.

Aside from the old "start merging modules" PITA that many of us are familiar with from the old "100 forward reference errors at the drop of a hat" days, I've found one viable (but still kinda PITA) strategy so far:

1. Look at the line that says: "module foo -> module bar -> module foo"

2. Pick one of those two modules (whichever has the simplest static ctors) and eliminate all the static ctors using the following pattern:

The pattern: The trick is to convert every variable that needs to be initialized into an "init on first use" ref @property. This "ref @property" checks a "hasThisBeenInited" bool and, if false, runs all the initialization. If the variable is a reference type, then *sometimes* you can get away with just checking for null (though often not, because it will just get-reinited ).

Example:
------------------------------
// Old:
class Foo { /+...+/ }
int number;
Foo foo;
static this
{
    foo = new Foo();
    number = /+ something maybe involving foo +/;
}

// New:
class Foo { /+...+/ }

private int _number;
@property ref int number()
{
    forceModuleInit();
    return _number;
}

private Foo _foo;
@property ref Foo foo()
{
    forceModuleInit();
    return _foo;
}

bool isModuleInited=false;
static void forceModuleInit() // Hopefully inlined
{
    if(!isModuleInited)
    {
        staticThis();
        isModuleInited = true;
    }
}
static void staticThis() // Workaround for static ctor cycle
{
    foo = new Foo();
    number = /+ something maybe involving foo +/;
}
------------------------------

If one of the variables being inited in the static ctor is something you *know* will never be accessed before some other specific variable is accessed, then you can skip converting it to "@property ref" if you want.

It's a big mess, but the conversion can be done deterministically, and even mechanically (heck, a ctfe string mixin wrapper could probably be built to do it).

The potential downsides:

1. If you come across an @propery bug or limitation, you're SOL. This should become less and less of an issue with time, though.

2. If one of the variables you converted is frequently-accessed, it could cause a performance problem.

3. Small increase to storage requirements. Might potentially be a problem if it's within templated or mixed-in code that gets instantiated many times.

At one point, I fiddled around with the idea of converting static ctors to "staticThis()" and then having one real static ctor for the entire library (assuming it's a library) that manually calls all the staticThis functions. One problem with this is that it's easy to accidentally forget to call one of the staticThis functions. The other big problem I found with this though, especially for a library, is that it requires everyone importing your code to always import through a single "import foo.all" module. If some user skips that, then the static ctors won't get run. There might be some possible workarounds for that, though:

- If the library has some primary interface that always gets used, then that can easily check if the static ctors have run and error out if not. If the primary interface is *always* the first part of your library used (or at least the first of all the parts that actually rely on the static ctors having run), then you could even run the static ctors right then instead of erroring out. That's a lot of "if"s, though, so it may not be widely-applicable.

- If you convert *all* static ctors to staticThis(), it might be possible to stick the one main static ctor into a private utility module that gets privately imported by all modules in the library. Then users can continue importing whatever module(s) they want. But if you don't convert *all* of the static ctors to staticThis, then you'll just re-introduce a cycle.

But if there's ever two separate libraries that have any interdependencies, then the one-main-real static ctor (that calls all the staticThis() funcs) will have to be shared between the two libraries. So overall, this approach may be possible, but maybe only in certain cases, and can involve a lot of changes.

"Nick Sabalausky" <a@a.a> wrote in message news:im8pmp$18p7$1@digitalmars.com...
>
> The pattern: The trick is to convert every variable that needs to be initialized into an "init on first use" ref @property. This "ref @property" checks a "hasThisBeenInited" bool and, if false, runs all the initialization. If the variable is a reference type, then *sometimes* you can get away with just checking for null (though often not, because it will just get-reinited ).
>

...if anyone sets it to null.

(Forgot to finish that last sentence.)

On Tue, 22 Mar 2011 02:12:55 +0200, Nick Sabalausky <a@a.a> wrote:

> The question: What now? What strategies do people find useful for dealing
> with this? Any specific "first steps" to take? Best practices? Etc.

Your post doesn't seem to mention it, but how about converting the static ctors to initialization functions, and calling them from a single static ctor within the dependency loop?

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

On Tue, 22 Mar 2011 04:30:37 +0200, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:

> Your post doesn't seem to mention it,

Sorry, didn't scroll down enough :)

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

"Vladimir Panteleev" <vladimir@thecybershadow.net> wrote in message news:op.vsp3zooituzx1w@cybershadow.mshome.net...
> On Tue, 22 Mar 2011 04:30:37 +0200, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:
>
>> Your post doesn't seem to mention it,
>
> Sorry, didn't scroll down enough :)
>

Well, that is a lot of scrolling, actually :)

On 03/22/2011 01:12 AM, Nick Sabalausky wrote:
> I'm intending this thread as somewhat of a roundtable-like discussion.
> Hopefully we can come up with good material for a short article on Wiki4D,
> or maybe the D website, or wherever.
>
> The scenario: A coder is writing some D, compiles, runs and gets a "Cyclic
> dependency in static ctors" error. Crap! A pain for experienced D users, and
> very bad PR for new D users. (Hopefully we'll eventually get some sort of
> solution for this, but in the meantime D users just have to deal with it.)
>
> The question: What now? What strategies do people find useful for dealing
> with this? Any specific "first steps" to take? Best practices? Etc.
>
> Aside from the old "start merging modules" PITA that many of us are familiar
> with from the old "100 forward reference errors at the drop of a hat" days,
> I've found one viable (but still kinda PITA) strategy so far:
>
> 1. Look at the line that says: "module foo ->  module bar ->  module foo"
>
> 2. Pick one of those two modules (whichever has the simplest static ctors)
> and eliminate all the static ctors using the following pattern:
>
> The pattern: The trick is to convert every variable that needs to be
> initialized into an "init on first use" ref @property. This "ref @property"
> checks a "hasThisBeenInited" bool and, if false, runs all the
> initialization. If the variable is a reference type, then *sometimes* you
> can get away with just checking for null (though often not, because it will
> just get-reinited ).
>
> Example:
> ------------------------------
> // Old:
> class Foo { /+...+/ }
> int number;
> Foo foo;
> static this
> {
>      foo = new Foo();
>      number = /+ something maybe involving foo +/;
> }
>
> // New:
> class Foo { /+...+/ }
>
> private int _number;
> @property ref int number()
> {
>      forceModuleInit();
>      return _number;
> }
>
> private Foo _foo;
> @property ref Foo foo()
> {
>      forceModuleInit();
>      return _foo;
> }
>
> bool isModuleInited=false;
> static void forceModuleInit() // Hopefully inlined
> {
>      if(!isModuleInited)
>      {
>          staticThis();
>          isModuleInited = true;
>      }
> }
> static void staticThis() // Workaround for static ctor cycle
> {
>      foo = new Foo();
>      number = /+ something maybe involving foo +/;
> }
> ------------------------------
>
> If one of the variables being inited in the static ctor is something you
> *know* will never be accessed before some other specific variable is
> accessed, then you can skip converting it to "@property ref" if you want.
>
> It's a big mess, but the conversion can be done deterministically, and even
> mechanically (heck, a ctfe string mixin wrapper could probably be built to
> do it).
>
> The potential downsides:
>
> 1. If you come across an @propery bug or limitation, you're SOL. This should
> become less and less of an issue with time, though.
>
> 2. If one of the variables you converted is frequently-accessed, it could
> cause a performance problem.
>
> 3. Small increase to storage requirements. Might potentially be a problem if
> it's within templated or mixed-in code that gets instantiated many times.
>
> At one point, I fiddled around with the idea of converting static ctors to
> "staticThis()" and then having one real static ctor for the entire library
> (assuming it's a library) that manually calls all the staticThis functions.
> One problem with this is that it's easy to accidentally forget to call one
> of the staticThis functions. The other big problem I found with this though,
> especially for a library, is that it requires everyone importing your code
> to always import through a single "import foo.all" module. If some user
> skips that, then the static ctors won't get run. There might be some
> possible workarounds for that, though:
>
> - If the library has some primary interface that always gets used, then that
> can easily check if the static ctors have run and error out if not. If the
> primary interface is *always* the first part of your library used (or at
> least the first of all the parts that actually rely on the static ctors
> having run), then you could even run the static ctors right then instead of
> erroring out. That's a lot of "if"s, though, so it may not be
> widely-applicable.
>
> - If you convert *all* static ctors to staticThis(), it might be possible to
> stick the one main static ctor into a private utility module that gets
> privately imported by all modules in the library. Then users can continue
> importing whatever module(s) they want. But if you don't convert *all* of
> the static ctors to staticThis, then you'll just re-introduce a cycle.
>
> But if there's ever two separate libraries that have any interdependencies,
> then the one-main-real static ctor (that calls all the staticThis() funcs)
> will have to be shared between the two libraries. So overall, this approach
> may be possible, but maybe only in certain cases, and can involve a lot of
> changes.

I think the idea of a single static constructor in a main lib import module (let's call it lib.d), calling staticThis func in every module, is the right track. First, it is rather a good practice (I mean both from the designer and user points of view).
Upon "it's easy to accidentally forget to call one of the staticThis functions": just systematically write one in every module, possibly empty at start. Then, from the main module, call staticThis on every imported module. This can be a cascade: staticThis if M calls staticThis of its own private dependencies. Shared submodules should still be init from lib, but even double init would be cheap thank the the bool flag.

Upon the case where users may need and import only part of your lib, then I guess obviously this mean some kind of *independency*, doesn't it? Else, all must be init-ed, and thus imported, anyway, don't you think? In which case they could as well import the main module in every case, and use only what they need.

Upon the last issue of common dependancies. If the problem of cyclic dependencies in static ctors is analog to circular imports, then there are 2 common strategies (here considering 2 modules importing each other):
* Isolate a part of a module that requires its own module and another one, place it in a 3rd module which imports both. (This is a common issue for test cases.)
* Conversely, isolate a part of a module that /is/ required by its own module and another one, place it in a 3rd module imported by both. (This is a common issue for tool features.)
Strangely enough, this often solves the problem by splitting further instead of by merging. Though I don't know whether this is appropriate for your issue.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com

On 03/22/2011 02:12 AM, Nick Sabalausky wrote:
> I'm intending this thread as somewhat of a roundtable-like discussion.
> Hopefully we can come up with good material for a short article on Wiki4D,
> or maybe the D website, or wherever.
>
> The scenario: A coder is writing some D, compiles, runs and gets a "Cyclic
> dependency in static ctors" error. Crap! A pain for experienced D users, and
> very bad PR for new D users. (Hopefully we'll eventually get some sort of
> solution for this, but in the meantime D users just have to deal with it.)
>
> The question: What now? What strategies do people find useful for dealing
> with this? Any specific "first steps" to take? Best practices? Etc.

One commonly used hack is to move static constructors into a separate helper module and call the initialization function via a C extern (like it is done in std.stdiobase):

----
module foo_helper;

private extern(C) foo_static_ctor();
static this()
{
    foo_static_ctor();
}

-----

module foo;
import foo_helper;

private Object global;
private extern(C) void foo_static_ctor()
{
    global = new Object;
}
----

Note that "global" is guaranteed to have been initialized when accessed from static constructors in modules that import "a". Being able to instruct the compiler to do this implicitly (so we could put static ctors in templates, for example) would probably solve most of static ctor problems.

>
> Aside from the old "start merging modules" PITA that many of us are familiar
> with from the old "100 forward reference errors at the drop of a hat" days,
> I've found one viable (but still kinda PITA) strategy so far:
>
> 1. Look at the line that says: "module foo ->  module bar ->  module foo"
>
> 2. Pick one of those two modules (whichever has the simplest static ctors)
> and eliminate all the static ctors using the following pattern:
>
> The pattern: The trick is to convert every variable that needs to be
> initialized into an "init on first use" ref @property. This "ref @property"
> checks a "hasThisBeenInited" bool and, if false, runs all the
> initialization. If the variable is a reference type, then *sometimes* you
> can get away with just checking for null (though often not, because it will
> just get-reinited ).
>
> Example:
> ------------------------------
> // Old:
> class Foo { /+...+/ }
> int number;
> Foo foo;
> static this
> {
>      foo = new Foo();
>      number = /+ something maybe involving foo +/;
> }
>
> // New:
> class Foo { /+...+/ }
>
> private int _number;
> @property ref int number()
> {
>      forceModuleInit();
>      return _number;
> }
>
> private Foo _foo;
> @property ref Foo foo()
> {
>      forceModuleInit();
>      return _foo;
> }
>
> bool isModuleInited=false;
> static void forceModuleInit() // Hopefully inlined
> {
>      if(!isModuleInited)
>      {
>          staticThis();
>          isModuleInited = true;
>      }
> }
> static void staticThis() // Workaround for static ctor cycle
> {
>      foo = new Foo();
>      number = /+ something maybe involving foo +/;
> }
> ------------------------------
>
> If one of the variables being inited in the static ctor is something you
> *know* will never be accessed before some other specific variable is
> accessed, then you can skip converting it to "@property ref" if you want.
>
> It's a big mess, but the conversion can be done deterministically, and even
> mechanically (heck, a ctfe string mixin wrapper could probably be built to
> do it).
>
> The potential downsides:
>
> 1. If you come across an @propery bug or limitation, you're SOL. This should
> become less and less of an issue with time, though.
>
> 2. If one of the variables you converted is frequently-accessed, it could
> cause a performance problem.
>
> 3. Small increase to storage requirements. Might potentially be a problem if
> it's within templated or mixed-in code that gets instantiated many times.

4. Initializing shared data needs synchronization. Then, your example would look similar to this:

------------------------------

class Foo { /+...+/ }

private immutable int _number;
@property ref int number()
{
     forceModuleInit();
     return _number;
}

private immutable Foo _foo;
@property immutable(Foo) foo()
{
     forceModuleInit();
     return _foo;
}

bool isModuleInited=false;
__gshared bool isSharedModuleInited=false;
static void forceModuleInit() // Hopefully inlined
{
     if(!isModuleInited)
     {
         synchronized(someLock)
         {
             if (!isSharedModuleInited)
             {
                 staticThis();
                 isSharedModuleInited = true;
             }
         }
         isModuleInited = true;
     }
}
static void sharedStaticThis() // Workaround for shared static ctor cycle
{
     auto foo = new Foo();
     _number = /+ something maybe involving foo +/;
     _foo = cast(immutable)foo;
}
------------------------------

>
> At one point, I fiddled around with the idea of converting static ctors to
> "staticThis()" and then having one real static ctor for the entire library
> (assuming it's a library) that manually calls all the staticThis functions.
> One problem with this is that it's easy to accidentally forget to call one
> of the staticThis functions. The other big problem I found with this though,
> especially for a library, is that it requires everyone importing your code
> to always import through a single "import foo.all" module. If some user
> skips that, then the static ctors won't get run. There might be some
> possible workarounds for that, though:
>
> - If the library has some primary interface that always gets used, then that
> can easily check if the static ctors have run and error out if not. If the
> primary interface is *always* the first part of your library used (or at
> least the first of all the parts that actually rely on the static ctors
> having run), then you could even run the static ctors right then instead of
> erroring out. That's a lot of "if"s, though, so it may not be
> widely-applicable.
>
> - If you convert *all* static ctors to staticThis(), it might be possible to
> stick the one main static ctor into a private utility module that gets
> privately imported by all modules in the library. Then users can continue
> importing whatever module(s) they want. But if you don't convert *all* of
> the static ctors to staticThis, then you'll just re-introduce a cycle.
>
> But if there's ever two separate libraries that have any interdependencies,
> then the one-main-real static ctor (that calls all the staticThis() funcs)
> will have to be shared between the two libraries. So overall, this approach
> may be possible, but maybe only in certain cases, and can involve a lot of
> changes.
>
>
>

On 2011-03-22 01:12, Nick Sabalausky wrote:
> At one point, I fiddled around with the idea of converting static ctors to
> "staticThis()" and then having one real static ctor for the entire library
> (assuming it's a library) that manually calls all the staticThis functions.
> One problem with this is that it's easy to accidentally forget to call one
> of the staticThis functions. The other big problem I found with this though,
> especially for a library, is that it requires everyone importing your code
> to always import through a single "import foo.all" module. If some user
> skips that, then the static ctors won't get run. There might be some
> possible workarounds for that, though:

One idea could be, although very platform dependent, to iterate the symbol table, finding all "staticThis" functions and calling them. Then you don't have the problem of forgetting to call one of those functions.

-- 
/Jacob Carlborg

> On 03/22/2011 02:12 AM, Nick Sabalausky wrote:
> > I'm intending this thread as somewhat of a roundtable-like discussion. Hopefully we can come up with good material for a short article on Wiki4D, or maybe the D website, or wherever.
> > 
> > The scenario: A coder is writing some D, compiles, runs and gets a "Cyclic dependency in static ctors" error. Crap! A pain for experienced D users, and very bad PR for new D users. (Hopefully we'll eventually get some sort of solution for this, but in the meantime D users just have to deal with it.)
> > 
> > The question: What now? What strategies do people find useful for dealing with this? Any specific "first steps" to take? Best practices? Etc.
> 
> One commonly used hack is to move static constructors into a separate helper module and call the initialization function via a C extern (like it is done in std.stdiobase):

That's what Phobos does to solve the problem (std.stdiobase being only one of the places that it does it). It's likely the solution that I would use as well.

- Jonathan M Davis

On 2011-03-22 05:16:31 -0400, Max Samukha <max@spam.box> said:

> ----
> module foo_helper;
> 
> private extern(C) foo_static_ctor();
> static this()
> {
>      foo_static_ctor();
> }
> 
> -----
> 
> module foo;
> import foo_helper;
> 
> private Object global;
> private extern(C) void foo_static_ctor()
> {
>      global = new Object;
> }
> ----
> 
> Note that "global" is guaranteed to have been initialized when accessed from static constructors in modules that import "a".

I don't know why people keep repeating that falacy. This statement is true only as long as there are no circular dependencies. It should read: "'global' is guarentied to have been initialized when access from static constructors in module that import 'a' _and_ which are not imported by 'a', directly or indirectly."

Because once you introduce a circular dependency, you get this:

----
module foo_helper;

private extern(C) foo_static_ctor();
static this() { foo_static_ctor(); }
-----
module foo;
import foo_helper;
import bar;

private Object global;
private extern(C) void foo_static_ctor()
{
    global = new Object;
    bar.testGlobal();
}
public void testGlobal()
{
    assert(global);
}
----
module bar_helper;

private extern(C) bar_static_ctor();
static this() { bar_static_ctor(); }
-----
module bar;
import bar_helper;
import foo;

private Object global;
private extern(C) void bar_static_ctor()
{
    global = new Object;
    foo.testGlobal();
}
public void testGlobal()
{
    assert(global);
}
----

Note how foo_static_ctor() and bar_static_ctor() each calls a function that needs the global variable of the other module to be initialized. It should be obvious that it can't work. If you doubt me, try it.


> Being able to instruct the compiler to do this implicitly (so we could put static ctors in templates, for example) would probably solve most of static ctor problems.

It'd have the exact same effect as adding a pragma to bypass the check for circular dependencies, while making things more complicated.


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/