On 15 September 2011 08:29, Rainer Schuetze <r.sagitario at gmx.de> wrote:
> On 15.09.2011 05:58, Benjamin Shropshire wrote:
>>
>> On 09/14/2011 12:06 AM, Rainer Schuetze wrote:
>>>
>>> On 14.09.2011 00:04, Walter Bright wrote:
>>>>
>>>> Don, just so you know, I've been thinking for a while about
>>>> transitioning from doing the semantic pass in order to doing it completely
>>>> "on demand". In other words, try to semantic a declaration. In the process,
>>>> any declarations it depends on are semantic'd if not already, recursively.
>>>
>>> I've been trying something similar for Visual D in its yet to integrate semantic analysis for intellisense. Still, static if and mixins get in the way of complete "on demand" handling. When a symbol is searched in a scope (e.g. a module, class, struct), some preparational work has to be done before the member list can be searched:
>>>
>>> 1. all "simple" non-scoping members are expanded (version/debug
>>> conditions, attributed declaration blocks). the branch inserted to the
>>> scopes' member list is also searched for "simple" non-scoping members (this
>>> step could also be done non-lazily, but doing it lazily slightly changes the
>>> interaction of version statements and conditionals with "static if"
>>> conditionals - good or bad, I don't know)
>>>
>>> 2. "complex" non-scoping members are expanded in lexical order (static
>>> if, mixins). When inserting the expanded branch into the scopes member list,
>>> the expansion restarts at 1.
>>>
>>> This works out better than the current dmd implementation, e.g. when forward referencing symbols in a mixin. There are still situations that depend on interpretation order, but that is to be expected when "static if" is used.
>>>
>>
>> Every time I've puzzled over the problem, the solution I've gravitated to is to have the symbol table logic result be tri-state: symbol-found, no-symbol, unknown/incomplete (for when a lookup includes an unprocessed scope). From there, you greedily evaluate all symbol that you can and proceed with whatever processing can be done, bailing when an "incomplete" results is found and keeping a list of where to come back and try again later. The only question then is how to handle the case where you dead lock. I suspect that if you make that illegal, a lot of legacy code will break. I'm going to guess we will want to have a small set of well thought out deadlock escape rules.
>
> I guess, Walter also wants to get rid of the "try again later" part. Especially is-expressions and trait(compiles) are getting rather indeterministic and might depend on other symbols being looked up before. If the evaluation is lazy, there is some hope that dependencies will be cyclic only in cases that are actual errors. I'm not sure about a good cycle-detection, though. (Is that what you meant by "deadlock escape"?)

trySemantic is the big problem. The traits(compiles) thing is only complicated because gagging is also used for trySemantic.

When traits(compiles) hits a forward reference, the correct logic is:
* if it is part of a speculative template instantiation, run semantic
on it with errors gagged;
* else, run normal semantic on it, without errors gagged.

But trySemantic is a big mess. To work properly it would need to be able to undo _everything_, and I don't think that's feasible -- I don't think we'd ever eliminate all the obscure bugs.

OTOH detecting cycles reliably in all cases might be nearly as difficult!

> What happens, if the evaluation of "static if" turns out to require symbols from the same scope? (Something I did not mention above: unconditionally existing or expanded members of a scope should be added to the symbol lookup as soon as possible.) My current suggestion is: do not recurse into the expansion of "complex" members, just use the currently available symbols.

What do you mean by "complex" members?

On 15.09.2011 09:00, Don Clugston wrote:
> On 15 September 2011 08:29, Rainer Schuetze<r.sagitario at gmx.de>  wrote:
>> What happens, if the evaluation of "static if" turns out to require symbols from the same scope? (Something I did not mention above: unconditionally existing or expanded members of a scope should be added to the symbol lookup as soon as possible.) My current suggestion is: do not recurse into the expansion of "complex" members, just use the currently available symbols.
> What do you mean by "complex" members?

"complex" in the sense of my (no longer quoted) initial mail: the members that have to be expanded before searching the scope is normally possible, i.e. static-if and mixins. They need semantic analysis before they might expand to new members that add more symbols to the scope.

On 15 September 2011 09:11, Rainer Schuetze <r.sagitario at gmx.de> wrote:
> On 15.09.2011 09:00, Don Clugston wrote:
>>
>> On 15 September 2011 08:29, Rainer Schuetze<r.sagitario at gmx.de> ?wrote:
>>>
>>> What happens, if the evaluation of "static if" turns out to require
>>> symbols
>>> from the same scope? (Something I did not mention above: unconditionally
>>> existing or expanded members of a scope should be added to the symbol
>>> lookup
>>> as soon as possible.) My current suggestion is: do not recurse into the
>>> expansion of "complex" members, just use the currently available symbols.
>>
>> What do you mean by "complex" members?
>
> "complex" in the sense of my (no longer quoted) initial mail: the members that have to be expanded before searching the scope is normally possible, i.e. static-if and mixins. They need semantic analysis before they might expand to new members that add more symbols to the scope.

I think there is a simple solution to 'static if'.
Do const folding on every static if condition, giving a boolean result.
Do NOT evaluate any static if bodies yet.
This would have the same effect as if every 'static if' were evaluated
simultaneously.
(There's a little bit of complexity with is() expressions that create
symbols, they need to be cached but not added to the scope yet).

Possibly const-fold every string mixin as well; this would make them
also act as if they were all evaluated simultaneously.
I don't know what should be done about template mixins though. But
they might not be much of a problem.
When this is complete, add all declarations from each static if body,
and from every mixin, into the scope.
When they're all inserted, look for static ifs and mixins again.
Repeat the process until none are left.

Am 15.09.2011, 11:44 Uhr, schrieb Don Clugston <dclugston at googlemail.com>:
> I think there is a simple solution to 'static if'.
> Do const folding on every static if condition, giving a boolean result.
> Do NOT evaluate any static if bodies yet.
> This would have the same effect as if every 'static if' were evaluated
> simultaneously.

What if the condition includes symbols from another static if's body or mixin or whatever?

On 15 September 2011 13:14,  <mrmocool at gmx.de> wrote:
> Am 15.09.2011, 11:44 Uhr, schrieb Don Clugston <dclugston at googlemail.com>:
>>
>> I think there is a simple solution to 'static if'.
>> Do const folding on every static if condition, giving a boolean result.
>> Do NOT evaluate any static if bodies yet.
>> This would have the same effect as if every 'static if' were evaluated
>> simultaneously.
>
> What if the condition includes symbols from another static if's body or mixin or whatever?

I think that should be disallowed.

On 9/15/11 4:44 AM, Don Clugston wrote:
> I think there is a simple solution to 'static if'.
> Do const folding on every static if condition, giving a boolean result.
> Do NOT evaluate any static if bodies yet.
> This would have the same effect as if every 'static if' were evaluated
> simultaneously.
> (There's a little bit of complexity with is() expressions that create
> symbols, they need to be cached but not added to the scope yet).
>
> Possibly const-fold every string mixin as well; this would make them
> also act as if they were all evaluated simultaneously.
> I don't know what should be done about template mixins though. But
> they might not be much of a problem.
> When this is complete, add all declarations from each static if body,
> and from every mixin, into the scope.
> When they're all inserted, look for static ifs and mixins again.
> Repeat the process until none are left.

This sounds like a very robust solution to me. It also produces intuitive behavior.

Andrei

On 9/15/11 6:53 AM, Don Clugston wrote:
> On 15 September 2011 13:14,<mrmocool at gmx.de>  wrote:
>> Am 15.09.2011, 11:44 Uhr, schrieb Don Clugston<dclugston at googlemail.com>:
>>>
>>> I think there is a simple solution to 'static if'.
>>> Do const folding on every static if condition, giving a boolean result.
>>> Do NOT evaluate any static if bodies yet.
>>> This would have the same effect as if every 'static if' were evaluated
>>> simultaneously.
>>
>> What if the condition includes symbols from another static if's body or mixin or whatever?
>
> I think that should be disallowed.

I see an issue here with cross-module use. For example, it's nice to have:

import some.module;

static if (is(typeof(some.module.foobar) == int)) {
    alias some.module.foobar baz;
} else {
    enum baz = 42; // or whatever
}

So far so good. The problem now is that some.module uses a similar technique to introduce that symbol foobar, the code won't work anymore.

I also realized that code relying on enumerating symbols in a module (like benchmark does) or a class (like an introspection library does) will miss all symbols guarded by static if. And, for example, ranges define plenty of those. This erodes the power of static if substantially.

I'm afraid it's back to the drawing board.


Andrei

On 15.09.2011 17:56, Andrei Alexandrescu wrote:
> On 9/15/11 6:53 AM, Don Clugston wrote:
>> On 15 September 2011 13:14,<mrmocool at gmx.de>  wrote:
>>> Am 15.09.2011, 11:44 Uhr, schrieb Don Clugston<dclugston at googlemail.com>:
>>>>
>>>> I think there is a simple solution to 'static if'.
>>>> Do const folding on every static if condition, giving a boolean
>>>> result.
>>>> Do NOT evaluate any static if bodies yet.
>>>> This would have the same effect as if every 'static if' were evaluated
>>>> simultaneously.
>>>
>>> What if the condition includes symbols from another static if's body or mixin or whatever?
>>
>> I think that should be disallowed.
>
> I see an issue here with cross-module use. For example, it's nice to have:
>
> import some.module;
>
> static if (is(typeof(some.module.foobar) == int)) {
>    alias some.module.foobar baz;
> } else {
>    enum baz = 42; // or whatever
> }
>
> So far so good. The problem now is that some.module uses a similar technique to introduce that symbol foobar, the code won't work anymore.
>

There is a much easier example against "simultaneous" static-if evaluation:

static if(size_t.sizeof == 8)
     enum is64bit = true;
else
     enum is64bit = false;

// later in the same module:
static if(is64bit)
{
}

will not compile because is64bit is not defined while all static ifs are evaluated.

> I also realized that code relying on enumerating symbols in a module (like benchmark does) or a class (like an introspection library does) will miss all symbols guarded by static if. And, for example, ranges define plenty of those. This erodes the power of static if substantially.
>

My proposal does not have this problem: all static ifs and mixins at the level of the scope are expanded in lexical order before it is searched (including enumeration). Problems might arise with circular symbol definitions: If evaluating the condition needs lookup of a symbol in the same scope, should that lookup force expanding later static-ifs and mixins or should it just work with the symbols available when starting evaluating the initial static-if? I think the latter seems better, because it is easier to understand.

class X
{
     static if(A.sizeof > 4) // will result in error if "static if(1)"
below is not expanded during evaluation
         int a;

     static if(1)   // should this static-if be expanded while
evaluating A.sizeof?
         enum N = 1;
     else
         enum N = 2;

     alias int[N] A; // error: undefined "N"?
}

There are corner cases that fail with this approach, but might compile with the "try again" approach, though with almost unpredictable results:

class C
{
     static if(is(typeof(A)))
         alias int B;
     static if(!is(typeof(B)))
         alias int A;

     B b;
}

On 09/14/2011 11:29 PM, Rainer Schuetze wrote:
> On 15.09.2011 05:58, Benjamin Shropshire wrote:
>> On 09/14/2011 12:06 AM, Rainer Schuetze wrote:
>>>
>>> On 14.09.2011 00:04, Walter Bright wrote:
>>>> Don, just so you know, I've been thinking for a while about transitioning from doing the semantic pass in order to doing it completely "on demand". In other words, try to semantic a declaration. In the process, any declarations it depends on are semantic'd if not already, recursively.
>>>
>>> I've been trying something similar for Visual D in its yet to integrate semantic analysis for intellisense. Still, static if and mixins get in the way of complete "on demand" handling. When a symbol is searched in a scope (e.g. a module, class, struct), some preparational work has to be done before the member list can be searched:
>>>
>>> 1. all "simple" non-scoping members are expanded (version/debug conditions, attributed declaration blocks). the branch inserted to the scopes' member list is also searched for "simple" non-scoping members (this step could also be done non-lazily, but doing it lazily slightly changes the interaction of version statements and conditionals with "static if" conditionals - good or bad, I don't know)
>>>
>>> 2. "complex" non-scoping members are expanded in lexical order (static if, mixins). When inserting the expanded branch into the scopes member list, the expansion restarts at 1.
>>>
>>> This works out better than the current dmd implementation, e.g. when forward referencing symbols in a mixin. There are still situations that depend on interpretation order, but that is to be expected when "static if" is used.
>>>
>>
>> Every time I've puzzled over the problem, the solution I've gravitated to is to have the symbol table logic result be tri-state: symbol-found, no-symbol, unknown/incomplete (for when a lookup includes an unprocessed scope). From there, you greedily evaluate all symbol that you can and proceed with whatever processing can be done, bailing when an "incomplete" results is found and keeping a list of where to come back and try again later. The only question then is how to handle the case where you dead lock. I suspect that if you make that illegal, a lot of legacy code will break. I'm going to guess we will want to have a small set of well thought out deadlock escape rules.
>
> I guess, Walter also wants to get rid of the "try again later" part. Especially is-expressions and trait(compiles) are getting rather indeterministic and might depend on other symbols being looked up before. If the evaluation is lazy, there is some hope that dependencies will be cyclic only in cases that are actual errors. I'm not sure about a good cycle-detection, though. (Is that what you meant by "deadlock escape"?)

Not exactly, what I'm thinking of wouldn't come into play until after a cycle has been detected. Breaking or escaping a deadlock would require ignoring an "unknown" lookup result by assuming the related scope(s) won't have any more relevant symbols. The issue is choosing where to make the guess. The simplest case would be a static if where you can look at both sides and see all the symbols (no complex mixins) and can show that regardless of which side is used the injected symbol won't effect it self.

>
> What happens, if the evaluation of "static if" turns out to require symbols from the same scope? (Something I did not mention above: unconditionally existing or expanded members of a scope should be added to the symbol lookup as soon as possible.) My current suggestion is: do not recurse into the expansion of "complex" members, just use the currently available symbols.
>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 15 September 2011 17:56, Andrei Alexandrescu <andrei at erdani.com> wrote:
> On 9/15/11 6:53 AM, Don Clugston wrote:
>>
>> On 15 September 2011 13:14,<mrmocool at gmx.de> ?wrote:
>>>
>>> Am 15.09.2011, 11:44 Uhr, schrieb Don Clugston<dclugston at googlemail.com>:
>>>>
>>>> I think there is a simple solution to 'static if'.
>>>> Do const folding on every static if condition, giving a boolean result.
>>>> Do NOT evaluate any static if bodies yet.
>>>> This would have the same effect as if every 'static if' were evaluated
>>>> simultaneously.
>>>
>>> What if the condition includes symbols from another static if's body or mixin or whatever?
>>
>> I think that should be disallowed.
>
> I see an issue here with cross-module use. For example, it's nice to have:
>
> import some.module;
>
> static if (is(typeof(some.module.foobar) == int)) {
> ? alias some.module.foobar baz;
> } else {
> ? enum baz = 42; // or whatever
> }
>
> So far so good. The problem now is that some.module uses a similar technique to introduce that symbol foobar, the code won't work anymore.
>
> I also realized that code relying on enumerating symbols in a module (like benchmark does) or a class (like an introspection library does) will miss all symbols guarded by static if. And, for example, ranges define plenty of those. This erodes the power of static if substantially.

Not so. The thing is, static ifs can be nested. Only one level of
static if is removed at a time.
If you simply wrap the static if inside static if(true) {...}
it won't be evaluated until all the first-level static ifs have added
their symbols to the scope.

So the current:
static if (cond1)
{
    A;
}
static if (cond2)
{
   B;
}
where cond2 depends on A, can be rewritten in the 'parallel execution'
paradigm as:
static if (cond1)
{
    A;
}
static if (true)
{
  static if (cond2)
  {
     B;
  }
}
Order of execution is controlled by depth of nesting, instead of by
order in the file.
Note that in your first example, the question of which module
instantiates the symbol is determined not even by order within the
file, but by which module is first on the command line -- ie, it's
determined by the makefile!