Not really understanding all the issues here, but this looks horrible :(

I hope this isn't the solution.

-Steve




>________________________________
>From: Don Clugston <dclugston at googlemail.com>
>To: Discuss the internals of DMD <dmd-internals at puremagic.com>
>Sent: Friday, September 16, 2011 2:04 AM
>Subject: Re: [dmd-internals] Fixing forward ref bugs for good
>
>On 15 September 2011 17:56, Andrei Alexandrescu <andrei at erdani.com> wrote:
>> On 9/15/11 6:53 AM, Don Clugston wrote:
>>>
>>> On 15 September 2011 13:14,<mrmocool at gmx.de> ?wrote:
>>>>
>>>> Am 15.09.2011, 11:44 Uhr, schrieb Don Clugston<dclugston at googlemail.com>:
>>>>>
>>>>> I think there is a simple solution to 'static if'.
>>>>> Do const folding on every static if condition, giving a boolean result.
>>>>> Do NOT evaluate any static if bodies yet.
>>>>> This would have the same effect as if every 'static if' were evaluated
>>>>> simultaneously.
>>>>
>>>> What if the condition includes symbols from another static if's body or mixin or whatever?
>>>
>>> I think that should be disallowed.
>>
>> I see an issue here with cross-module use. For example, it's nice to have:
>>
>> import some.module;
>>
>> static if (is(typeof(some.module.foobar) == int)) {
>> ? alias some.module.foobar baz;
>> } else {
>> ? enum baz = 42; // or whatever
>> }
>>
>> So far so good. The problem now is that some.module uses a similar technique to introduce that symbol foobar, the code won't work anymore.
>>
>> I also realized that code relying on enumerating symbols in a module (like benchmark does) or a class (like an introspection library does) will miss all symbols guarded by static if. And, for example, ranges define plenty of those. This erodes the power of static if substantially.
>
>Not so. The thing is, static ifs can be nested. Only one level of
>static if is removed at a time.
>If you simply wrap the static if inside static if(true) {...}
>it won't be evaluated until all the first-level static ifs have added
>their symbols to the scope.
>
>So the current:
>static if (cond1)
>{
>? ? A;
>}
>static if (cond2)
>{
>?  B;
>}
>where cond2 depends on A, can be rewritten in the 'parallel execution'
>paradigm as:
>static if (cond1)
>{
>? ? A;
>}
>static if (true)
>{
>? static if (cond2)
>? {
>? ?  B;
>? }
>}
>Order of execution is controlled by depth of nesting, instead of by
>order in the file.
>Note that in your first example, the question of which module
>instantiates the symbol is determined not even by order within the
>file, but by which module is first on the command line -- ie, it's
>determined by the makefile!
>_______________________________________________
>dmd-internals mailing list
>dmd-internals at puremagic.com
>http://lists.puremagic.com/mailman/listinfo/dmd-internals
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/dmd-internals/attachments/20110916/42565f6b/attachment-0001.html>

On 16 September 2011 13:25, Steve Schveighoffer <schveiguy at yahoo.com> wrote:
> Not really understanding all the issues here, but this looks horrible :(
> I hope this isn't the solution.
> -Steve

Are you reacting to the concept, or only to the syntax?
Bear in mind that it occurs ONLY when you have dependent static ifs
within a single scope.
For example, if you have a 'static if' inside a template, struct, or
function, it has access to ALL symbols in that module.
In my opinion, this normally (not always, but normally) indicates a
poor design. It can almost always be factored out (eg, by creating a
bool enum).  Worst case, we could add some syntax sugar.
I think that if you really want to have semantics which are
independent of the ordering of declarations, then by definition you
can't have dependent static ifs in the same scope.
The second option is to allow ordering to be significant only for
mixins and static if. And in that case, the ordering has to be strict:
even if something near the end of the file is 'static if(1)', the
declarations in it are not available anywhere else in the scope.
I think any other option (like evaluating every static if that doesn't
use is(typeof()) or other forms of reflection, before those that do)
leads to madness.


> From: Don Clugston <dclugston at googlemail.com>
> On 15 September 2011 17:56, Andrei Alexandrescu <andrei at erdani.com> wrote:
>> On 9/15/11 6:53 AM, Don Clugston wrote:
>>>
>>> On 15 September 2011 13:14,<mrmocool at gmx.de> ?wrote:
>>>>
>>>> Am 15.09.2011, 11:44 Uhr, schrieb Don
>>>> Clugston<dclugston at googlemail.com>:
>>>>>
>>>>> I think there is a simple solution to 'static if'.
>>>>> Do const folding on every static if condition, giving a boolean result.
>>>>> Do NOT evaluate any static if bodies yet.
>>>>> This would have the same effect as if every 'static if' were evaluated
>>>>> simultaneously.
>>>>
>>>> What if the condition includes symbols from another static if's body or mixin or whatever?
>>>
>>> I think that should be disallowed.
>>
>> I see an issue here with cross-module use. For example, it's nice to have:
>>
>> import some.module;
>>
>> static if (is(typeof(some.module.foobar) == int)) {
>> ? alias some.module.foobar baz;
>> } else {
>> ? enum baz = 42; // or whatever
>> }
>>
>> So far so good. The problem now is that some.module uses a similar
>> technique
>> to introduce that symbol foobar, the code won't work anymore.
>>
>> I also realized that code relying on enumerating symbols in a module (like
>> benchmark does) or a class (like an introspection library does) will miss
>> all symbols guarded by static if. And, for example, ranges define plenty
>> of
>> those. This erodes the power of static if substantially.
>
> Not so. The thing is, static ifs can be nested. Only one level of
> static if is removed at a time.
> If you simply wrap the static if inside static if(true) {...}
> it won't be evaluated until all the first-level static ifs have added
> their symbols to the scope.
>
> So the current:
> static if (cond1)
> {
> ? ? A;
> }
> static if (cond2)
> {
> ? B;
> }
> where cond2 depends on A, can be rewritten in the 'parallel execution'
> paradigm as:
> static if (cond1)
> {
> ? ? A;
> }
> static if (true)
> {
> ? static if (cond2)
> ? {
> ? ? B;
> ? }
> }
> Order of execution is controlled by depth of nesting, instead of by
> order in the file.
> Note that in your first example, the question of which module
> instantiates the symbol is determined not even by order within the
> file, but by which module is first on the command line -- ie, it's
> determined by the makefile!
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>
>
>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>

?Sorry, did not mean this to be private...



----- Forwarded Message -----
>From: Steve Schveighoffer <schveiguy at yahoo.com>
>To: Don Clugston <dclugston at googlemail.com>
>Sent: Friday, September 16, 2011 8:57 AM
>Subject: Re: [dmd-internals] Fixing forward ref bugs for good
>
>
>What I mean is, it feels like the wrong approach to ordering.? I'd rather ordering be dependent on ordering, not nesting.? It seems like a counter-intuitive way to do ordering that has the same issues using the order of declaration.
>
>
>
>People are going to look at some file that has static if(true) (or several levels of them!) and scratch their heads, potentially removing it.
>
>
>Are we trying to say here, ordering matters only for static ifs?? Then why can't the ordering just be the order they appear?? Is there any advantage to use nesting?? What about some sort of pragma or @annotation to define ordering?
>
>
>And what about repeatedly trying evaluating a static if until it can be evaluated, or you are stuck?? Has that been considered and rejected for some reason?
>
>
>Again, not understanding all the issues, so maybe it's just the only way possible.? But it sure looks bad.
>
>
>
>-Steve
>
>
>
>
>>________________________________
>>From: Don Clugston <dclugston at googlemail.com>
>>To: Steve Schveighoffer <schveiguy at yahoo.com>; Discuss the internals of DMD <dmd-internals at puremagic.com>
>>Sent: Friday, September 16, 2011 8:37 AM
>>Subject: Re: [dmd-internals] Fixing forward ref bugs for good
>>
>>On 16 September 2011 13:25, Steve Schveighoffer <schveiguy at yahoo.com> wrote:
>>> Not really understanding all the issues here, but this looks horrible :(
>>> I hope this isn't the solution.
>>> -Steve
>>
>>Are you reacting to the concept, or only to the syntax?
>>Bear in mind that it occurs ONLY when you have dependent static ifs
>>within a single scope.
>>For example, if you have a 'static if' inside a template, struct, or
>>function, it has access to ALL symbols in that module.
>>In my opinion, this normally (not always, but normally) indicates a
>>poor design. It can almost always be factored out (eg, by creating a
>>bool enum).? Worst case, we could add some syntax sugar.
>>I think that if
 you really want to have semantics which are
>>independent of the ordering of declarations, then by definition you
>>can't have dependent static ifs in the same scope.
>>The second option is to allow ordering to be significant only for
>>mixins and static if. And in that case, the ordering has to be strict:
>>even if something near the end of the file is 'static if(1)', the
>>declarations in it are not available anywhere else in the scope.
>>I think any other option (like evaluating every static if that doesn't
>>use is(typeof()) or other forms of reflection, before those that do)
>>leads to madness.
>>
>>
>>> From: Don Clugston <dclugston at googlemail.com>
>>> On 15 September 2011 17:56, Andrei Alexandrescu <andrei at erdani.com> wrote:
>>>> On 9/15/11 6:53 AM,
 Don Clugston wrote:
>>>>>
>>>>> On 15 September 2011 13:14,<mrmocool at gmx.de> ?wrote:
>>>>>>
>>>>>> Am 15.09.2011, 11:44 Uhr, schrieb Don
>>>>>> Clugston<dclugston at googlemail.com>:
>>>>>>>
>>>>>>> I think there is a simple solution to 'static if'.
>>>>>>> Do const folding on every static if condition, giving a boolean result.
>>>>>>> Do NOT evaluate any static if bodies yet.
>>>>>>> This would have the same effect as if every 'static if' were evaluated
>>>>>>> simultaneously.
>>>>>>
>>>>>> What if the condition includes symbols from another static if's body or mixin or
 whatever?
>>>>>
>>>>> I think that should be disallowed.
>>>>
>>>> I see an issue here with cross-module use. For example, it's nice to have:
>>>>
>>>> import some.module;
>>>>
>>>> static if (is(typeof(some.module.foobar) == int)) {
>>>> ? alias some.module.foobar baz;
>>>> } else {
>>>> ? enum baz = 42; // or whatever
>>>> }
>>>>
>>>> So far so good. The problem now is that some.module uses a similar
>>>> technique
>>>> to introduce that symbol foobar, the code won't work anymore.
>>>>
>>>> I also realized that code relying on enumerating symbols in a module (like
>>>> benchmark does) or a class (like an introspection library does) will miss
>>>> all symbols guarded by static if. And, for example, ranges define plenty
>>>> of
>>>> those. This erodes the power of static if
 substantially.
>>>
>>> Not so. The thing is, static ifs can be nested. Only one level of
>>> static if is removed at a time.
>>> If you simply wrap the static if inside static if(true) {...}
>>> it won't be evaluated until all the first-level static ifs have added
>>> their symbols to the scope.
>>>
>>> So the current:
>>> static if (cond1)
>>> {
>>> ? ? A;
>>> }
>>> static if (cond2)
>>> {
>>> ? B;
>>> }
>>> where cond2 depends on A, can be rewritten in the 'parallel execution'
>>> paradigm as:
>>> static if (cond1)
>>> {
>>> ? ? A;
>>> }
>>> static if (true)
>>> {
>>> ? static if (cond2)
>>> ? {
>>> ? ? B;
>>> ? }
>>> }
>>> Order of execution is controlled by depth of nesting, instead of by
>>> order in the file.
>>> Note that in your first example, the question of which
 module
>>> instantiates the symbol is determined not even by order within the
>>> file, but by which module is first on the command line -- ie, it's
>>> determined by the makefile!
>>> _______________________________________________
>>> dmd-internals mailing list
>>> dmd-internals at puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>>>
>>>
>>>
>>> _______________________________________________
>>> dmd-internals mailing list
>>> dmd-internals at puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>>>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/dmd-internals/attachments/20110916/02916f89/attachment.html>

> ----- Forwarded Message -----
> From: Steve Schveighoffer <schveiguy at yahoo.com>
> What I mean is, it feels like the wrong approach to ordering.? I'd rather
> ordering be dependent on ordering, not nesting.? It seems like a
> counter-intuitive way to do ordering that has the same issues using the
> order of declaration.
>
> People are going to look at some file that has static if(true) (or several
> levels of them!) and scratch their heads, potentially removing it.
> Are we trying to say here, ordering matters only for static ifs?

The problem isn't static if, or even ordering, per se. It's the
combination of (a) reflection, and (b) the fact that declarations can
be conditionally added via static if and mixin; the two concepts are
fundamentally incompatible.
Any reflection that checks for existence of a symbol has a result
which can change with time.

So, although the spec says that order of declarations doesn't matter, it isn't:

enum b1 = is(typeof(foo));
static if (!b2)  int foo;
enum b2 = is(typeof(foo));

Is b1 true, or false? Currently, it's false, but becomes true if moved to the bottom of the file.

You could say that any use of a non-existent symbol potentially
depends on all non-evaluated static ifs; you don't know it doesn't
exist until you've expanded them ALL.
So one idea would be to evaluate every static if that had a condition
that didn't involve reflection. If you find a condition which is
blocked, you stop, and go onto the next one. Moderately difficult to
implement, but possible.
But often you get multiple static ifs which are blocked. They are
probably independent, but you don't know. What do you do? Well, you
could arbitrarily say you do them in the order the static ifs appear
in the file.
But it's not very predictable, because you can't just look at a static
if statement and see if it is blocked or not -- the reflection could
be happening anywhere in the file. I don't think that's viable.

>  Then why
> can't the ordering just be the order they appear?  Is there any advantage to
> use nesting?

The nesting behaviour is a natural consequence of saying that the order doesn't matter. It's not a design decision. It does give an advantage over order-in-file in the case of things like struct members (where order in the file DOES matter) because you can control the order the static ifs are evaluated (you can make the first static if in the struct only be evaluated after all the others have finished).

>? What about some sort of pragma or @annotation to define
> ordering?

That's possible, too. (That's what I meant by syntax sugar).

> And what about repeatedly trying evaluating a static if until it can be evaluated, or you are stuck?? Has that been considered and rejected for some reason?

See above. It's not feasible.

> Again, not understanding all the issues, so maybe it's just the only way possible.? But it sure looks bad.

Fundamentally we're doing something which is contradictory (but so useful in practice that we desperately want to keep it!) I think a perfect solution is impossible.

I think we have four feasible solutions:
(1)
static ifs and mixins are expanded in the order they appear in the
file. This is applied recursively until none are left.
Finally, everything else is evaluated in parallel.
(2)
static ifs and mixins are expanded in parallel. This is repeated until
none are left.
Finally, everything else is evaluated in parallel.
(3)
 Everything is evaluated in parallel, except for static ifs and mixins.
 If no static ifs or mixins, quit.
 static ifs and mixins are evaluated & expanded in the order they
appear in the file. Repeat.
(4)
 Everything is evaluated in parallel, including static ifs and mixins
(but they aren't expanded).
 If no static ifs or mixins, quit.
static ifs and mixins are expanded. Repeat.

All of these have surprising behaviour in some ways.
-Don.

> From: Don Clugston <dclugston at googlemail.com>
> On 16 September 2011 13:25, Steve Schveighoffer <schveiguy at yahoo.com> wrote:
>> Not really understanding all the issues here, but this looks horrible :(
>> I hope this isn't the solution.
>> -Steve
>
> Are you reacting to the concept, or only to the syntax?
> Bear in mind that it occurs ONLY when you have dependent static ifs
> within a single scope.
> For example, if you have a 'static if' inside a template, struct, or
> function, it has access to ALL symbols in that module.
> In my opinion, this normally (not always, but normally) indicates a
> poor design. It can almost always be factored out (eg, by creating a
> bool enum).? Worst case, we could add some syntax sugar.
> I think that if you really want to have semantics which are
> independent of the ordering of declarations, then by definition you
> can't have dependent static ifs in the same scope.
> The second option is to allow ordering to be significant only for
> mixins and static if. And in that case, the ordering has to be strict:
> even if something near the end of the file is 'static if(1)', the
> declarations in it are not available anywhere else in the scope.
> I think any other option (like evaluating every static if that doesn't
> use is(typeof()) or other forms of reflection, before those that do)
> leads to madness.
>
>
>> From: Don Clugston <dclugston at googlemail.com>
>> On 15 September 2011 17:56, Andrei Alexandrescu <andrei at erdani.com> wrote:
>>> On 9/15/11 6:53 AM, Don Clugston wrote:
>>>>
>>>> On 15 September 2011 13:14,<mrmocool at gmx.de> ?wrote:
>>>>>
>>>>> Am 15.09.2011, 11:44 Uhr, schrieb Don
>>>>> Clugston<dclugston at googlemail.com>:
>>>>>>
>>>>>> I think there is a simple solution to 'static if'.
>>>>>> Do const folding on every static if condition, giving a boolean
>>>>>> result.
>>>>>> Do NOT evaluate any static if bodies yet.
>>>>>> This would have the same effect as if every 'static if' were evaluated
>>>>>> simultaneously.
>>>>>
>>>>> What if the condition includes symbols from another static if's body or mixin or whatever?
>>>>
>>>> I think that should be disallowed.
>>>
>>> I see an issue here with cross-module use. For example, it's nice to have:
>>>
>>> import some.module;
>>>
>>> static if (is(typeof(some.module.foobar) == int)) {
>>> ? alias some.module.foobar baz;
>>> } else {
>>> ? enum baz = 42; // or whatever
>>> }
>>>
>>> So far so good. The problem now is that some.module uses a similar
>>> technique
>>> to introduce that symbol foobar, the code won't work anymore.
>>>
>>> I also realized that code relying on enumerating symbols in a module
>>> (like
>>> benchmark does) or a class (like an introspection library does) will miss
>>> all symbols guarded by static if. And, for example, ranges define plenty
>>> of
>>> those. This erodes the power of static if substantially.
>>
>> Not so. The thing is, static ifs can be nested. Only one level of
>> static if is removed at a time.
>> If you simply wrap the static if inside static if(true) {...}
>> it won't be evaluated until all the first-level static ifs have added
>> their symbols to the scope.
>>
>> So the current:
>> static if (cond1)
>> {
>> ? ? A;
>> }
>> static if (cond2)
>> {
>> ? B;
>> }
>> where cond2 depends on A, can be rewritten in the 'parallel execution'
>> paradigm as:
>> static if (cond1)
>> {
>> ? ? A;
>> }
>> static if (true)
>> {
>> ? static if (cond2)
>> ? {
>> ? ? B;
>> ? }
>> }
>> Order of execution is controlled by depth of nesting, instead of by
>> order in the file.
>> Note that in your first example, the question of which module
>> instantiates the symbol is determined not even by order within the
>> file, but by which module is first on the command line -- ie, it's
>> determined by the makefile!
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>>
>>
>>
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>>
>
>
>
>
>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>

On 9/16/2011 1:03 PM, Don Clugston wrote:
>> ----- Forwarded Message -----
>> From: Steve Schveighoffer<schveiguy at yahoo.com>
>> What I mean is, it feels like the wrong approach to ordering.  I'd rather
>> ordering be dependent on ordering, not nesting.  It seems like a
>> counter-intuitive way to do ordering that has the same issues using the
>> order of declaration.
>>
>> People are going to look at some file that has static if(true) (or several
>> levels of them!) and scratch their heads, potentially removing it.
>> Are we trying to say here, ordering matters only for static ifs?
> The problem isn't static if, or even ordering, per se. It's the
> combination of (a) reflection, and (b) the fact that declarations can
> be conditionally added via static if and mixin; the two concepts are
> fundamentally incompatible.
> Any reflection that checks for existence of a symbol has a result
> which can change with time.
>
> So, although the spec says that order of declarations doesn't matter, it isn't:
>
> enum b1 = is(typeof(foo));
> static if (!b2)  int foo;
> enum b2 = is(typeof(foo));
>
> Is b1 true, or false? Currently, it's false, but becomes true if moved to the bottom of the file.
>
> You could say that any use of a non-existent symbol potentially
> depends on all non-evaluated static ifs; you don't know it doesn't
> exist until you've expanded them ALL.
> So one idea would be to evaluate every static if that had a condition
> that didn't involve reflection. If you find a condition which is
> blocked, you stop, and go onto the next one. Moderately difficult to
> implement, but possible.
> But often you get multiple static ifs which are blocked. They are
> probably independent, but you don't know. What do you do? Well, you
> could arbitrarily say you do them in the order the static ifs appear
> in the file.
> But it's not very predictable, because you can't just look at a static
> if statement and see if it is blocked or not -- the reflection could
> be happening anywhere in the file. I don't think that's viable.
>
>>   Then why
>> can't the ordering just be the order they appear?  Is there any advantage to
>> use nesting?
> The nesting behaviour is a natural consequence of saying that the order doesn't matter. It's not a design decision. It does give an advantage over order-in-file in the case of things like struct members (where order in the file DOES matter) because you can control the order the static ifs are evaluated (you can make the first static if in the struct only be evaluated after all the others have finished).
>
>>    What about some sort of pragma or @annotation to define
>> ordering?
> That's possible, too. (That's what I meant by syntax sugar).
>
>> And what about repeatedly trying evaluating a static if until it can be evaluated, or you are stuck?  Has that been considered and rejected for some reason?
> See above. It's not feasible.
>
>> Again, not understanding all the issues, so maybe it's just the only way possible.  But it sure looks bad.
> Fundamentally we're doing something which is contradictory (but so useful in practice that we desperately want to keep it!) I think a perfect solution is impossible.
>
> I think we have four feasible solutions:
> (1)
> static ifs and mixins are expanded in the order they appear in the
> file. This is applied recursively until none are left.
> Finally, everything else is evaluated in parallel.
> (2)
> static ifs and mixins are expanded in parallel. This is repeated until
> none are left.
> Finally, everything else is evaluated in parallel.
> (3)
>   Everything is evaluated in parallel, except for static ifs and mixins.
>   If no static ifs or mixins, quit.
>   static ifs and mixins are evaluated&  expanded in the order they
> appear in the file. Repeat.
> (4)
>   Everything is evaluated in parallel, including static ifs and mixins
> (but they aren't expanded).
>   If no static ifs or mixins, quit.
> static ifs and mixins are expanded. Repeat.
>
> All of these have surprising behaviour in some ways.
> -Don.
>

I agree with Don's compilation. I just want to point out that the
expansion does not need to recurse into nested scopes during the steps
above (if there are no explicite symbol lookups during condition/mixin
evaluation). All we want is the *complete* list of symbols in the
current scope.
Looking into nested scopes might get delayed by lazy evaluation, maybe
even skipped in a fast compilation mode.

On 16 sep 2011, at 22:03, Don Clugston wrote:
> Fundamentally we're doing something which is contradictory (but so useful in practice that we desperately want to keep it!) I think a perfect solution is impossible.
> 
> I think we have four feasible solutions:
> (1)
> static ifs and mixins are expanded in the order they appear in the
> file. This is applied recursively until none are left.
> Finally, everything else is evaluated in parallel.
> (2)
> static ifs and mixins are expanded in parallel. This is repeated until
> none are left.
> Finally, everything else is evaluated in parallel.
> (3)
> Everything is evaluated in parallel, except for static ifs and mixins.
> If no static ifs or mixins, quit.
> static ifs and mixins are evaluated & expanded in the order they
> appear in the file. Repeat.
> (4)
> Everything is evaluated in parallel, including static ifs and mixins
> (but they aren't expanded).
> If no static ifs or mixins, quit.
> static ifs and mixins are expanded. Repeat.
> 
> All of these have surprising behaviour in some ways.
> -Don.
> 


So which one of these is preferable?

-- 
/Jacob Carlborg

>________________________________
>From: Don Clugston <dclugston at googlemail.com>
>To: Discuss the internals of DMD <dmd-internals at puremagic.com>
>Sent: Friday, September 16, 2011 4:03 PM
>Subject: Re: [dmd-internals] Fw: Fixing forward ref bugs for good
>
>> ----- Forwarded Message -----
>> From: Steve Schveighoffer <schveiguy at yahoo.com>
>> What I mean is, it feels like the wrong approach to ordering.? I'd rather
>> ordering be dependent on ordering, not nesting.? It seems like a
>> counter-intuitive way to do ordering that has the same issues using the
>> order of declaration.
>>
>> People are going to look at some file that has static if(true) (or several
>> levels of them!) and scratch their heads, potentially removing it.
>> Are we trying to say here, ordering matters only for static ifs?
>
>The problem isn't static if, or even ordering, per se. It's the
>combination of (a) reflection, and (b) the fact that declarations can
>be conditionally added via static if and mixin; the two concepts are
>fundamentally incompatible.
>Any reflection that checks for existence of a symbol has a result
>which can change with time.
>
>So, although the spec says that order of declarations doesn't matter, it isn't:
>
>enum b1 = is(typeof(foo));
>static if (!b2)? int foo;
>enum b2 = is(typeof(foo));
>
>Is b1 true, or false? Currently, it's false, but becomes true if moved to the bottom of the file.

I understand the current issues are hairy, but I think understanding that the above b1 can change if it's moved to be later in the file is easier than understanding that:

static if(true)? enum b1 = is(typeof(foo));
static if(!b2) int foo;
enum b2 = is(typeof(foo));

the static if(true) makes a difference.? potentially, if something depends on b1, it needs 2 static ifs in order to be evaluated properly.

It seems like an "artifical" solution to being able to declare things out of oder -- you still need to follow some ordering, only now it's not sequential, it's hierarchical.? It's a lot easier for me to see what order things are evaluated when they are in some order in the file than it is to see the ordering from the level of nesting.

That's all I'm saying.? It's going to add a level of confusion, and it still seems like you need to be careful of ordering (nesting).

>The nesting behaviour is a natural consequence of saying that the

>order doesn't matter. It's not a design decision. It does give an advantage over order-in-file in the case of things like struct members (where order in the file DOES matter) because you can control the order the static ifs are evaluated (you can make the first static if in the struct only be evaluated after all the others have finished).

Yes, I can see why you'd want that, but that seems fringe-ish.? It's probably fixable by repeating some part of the code just for evaluating the static if, or factoring something out of the struct.

We should also be considering saying order *does* matter in certain cases.

>> And what about repeatedly trying evaluating a static if until it can be evaluated, or you are stuck?? Has that been considered and rejected for some reason?
>
>See above. It's not feasible.

I wasn't thinking of static ifs which depend on seeing if a symbol exists.? I agree those are potentially different depending on some ordering.

-Steve

On 17 September 2011 08:00, Rainer Schuetze <r.sagitario at gmx.de> wrote:
> On 9/16/2011 1:03 PM, Don Clugston wrote:
>>>
>>> ----- Forwarded Message -----
>>> From: Steve Schveighoffer<schveiguy at yahoo.com>
>>> What I mean is, it feels like the wrong approach to ordering. ?I'd rather
>>> ordering be dependent on ordering, not nesting. ?It seems like a
>>> counter-intuitive way to do ordering that has the same issues using the
>>> order of declaration.
>>>
>>> People are going to look at some file that has static if(true) (or
>>> several
>>> levels of them!) and scratch their heads, potentially removing it.
>>> Are we trying to say here, ordering matters only for static ifs?
>>
>> The problem isn't static if, or even ordering, per se. It's the
>> combination of (a) reflection, and (b) the fact that declarations can
>> be conditionally added via static if and mixin; the two concepts are
>> fundamentally incompatible.
>> Any reflection that checks for existence of a symbol has a result
>> which can change with time.
>>
>> So, although the spec says that order of declarations doesn't matter, it isn't:
>>
>> enum b1 = is(typeof(foo));
>> static if (!b2) ?int foo;
>> enum b2 = is(typeof(foo));
>>
>> Is b1 true, or false? Currently, it's false, but becomes true if moved to the bottom of the file.
>>
>> You could say that any use of a non-existent symbol potentially
>> depends on all non-evaluated static ifs; you don't know it doesn't
>> exist until you've expanded them ALL.
>> So one idea would be to evaluate every static if that had a condition
>> that didn't involve reflection. If you find a condition which is
>> blocked, you stop, and go onto the next one. Moderately difficult to
>> implement, but possible.
>> But often you get multiple static ifs which are blocked. They are
>> probably independent, but you don't know. What do you do? Well, you
>> could arbitrarily say you do them in the order the static ifs appear
>> in the file.
>> But it's not very predictable, because you can't just look at a static
>> if statement and see if it is blocked or not -- the reflection could
>> be happening anywhere in the file. I don't think that's viable.
>>
>>> ?Then why
>>> can't the ordering just be the order they appear? ?Is there any advantage
>>> to
>>> use nesting?
>>
>> The nesting behaviour is a natural consequence of saying that the order doesn't matter. It's not a design decision. It does give an advantage over order-in-file in the case of things like struct members (where order in the file DOES matter) because you can control the order the static ifs are evaluated (you can make the first static if in the struct only be evaluated after all the others have finished).
>>
>>> ? What about some sort of pragma or @annotation to define ordering?
>>
>> That's possible, too. (That's what I meant by syntax sugar).
>>
>>> And what about repeatedly trying evaluating a static if until it can be
>>> evaluated, or you are stuck? ?Has that been considered and rejected for
>>> some
>>> reason?
>>
>> See above. It's not feasible.
>>
>>> Again, not understanding all the issues, so maybe it's just the only way possible. ?But it sure looks bad.
>>
>> Fundamentally we're doing something which is contradictory (but so useful in practice that we desperately want to keep it!) I think a perfect solution is impossible.
>>
>> I think we have four feasible solutions:
>> (1)
>> static ifs and mixins are expanded in the order they appear in the
>> file. This is applied recursively until none are left.
>> Finally, everything else is evaluated in parallel.
>> (2)
>> static ifs and mixins are expanded in parallel. This is repeated until
>> none are left.
>> Finally, everything else is evaluated in parallel.
>> (3)
>> ?Everything is evaluated in parallel, except for static ifs and mixins.
>> ?If no static ifs or mixins, quit.
>> ?static ifs and mixins are evaluated& ?expanded in the order they
>> appear in the file. Repeat.
>> (4)
>> ?Everything is evaluated in parallel, including static ifs and mixins
>> (but they aren't expanded).
>> ?If no static ifs or mixins, quit.
>> static ifs and mixins are expanded. Repeat.
>>
>> All of these have surprising behaviour in some ways.
>> -Don.
>>
>
> I agree with Don's compilation. I just want to point out that the expansion
> does not need to recurse into nested scopes during the steps above (if there
> are no explicite symbol lookups during condition/mixin evaluation). All we
> want is the *complete* list of symbols in the current scope.
> Looking into nested scopes might get delayed by lazy evaluation, maybe even
> skipped in a fast compilation mode.

It looks as though we want a semantic pass 0, which evaluates all of
the mixins and static ifs.
Then the reflection issue is resolved as:
* any reflection which is evaluated while expanding a mixin or a
static if condition, represents an intermediate, potentially unstable
state, while symbols are still being generated.
* All other reflection represents the state after all symbols are added.

We can still have some pretty weird behaviour, where an expression is
used in a static if, but the same expression is different everywhere
else. Eg,
template Foo(int X)
{
     bool Foo = is(typeof(bar));
}
static if (!Foo!(2))  int bar;

static assert( Foo!(1) == Foo!(2) );  // fails!

We probably have to live with that.
The only alternative I can think of, is to dump the symbol table at
the end of the semantic0 pass, so that _all_ values are re-evaluated,
and so that you don't have to worry about whether something was used
in a static if, or not. This would be implemented, once again, by
storing the const-folding result into the mixins and static if's, and
then clearing the symbol table for that scope. This would mean that
static if and mixins are always 'snapshots' of the state of
compilation, but everything else represents the final state. With my
suggested 'do everything in parallel' you'd only have to clear the
symbol table once, but if they are evaluated in order, you need to
wipe the symbol table after every evaluation of a static if or mixin.
But the impact on compilation time could be kept small, by only wiping
the symbol table if you had a reflection result which might be
different to the final one; ie, if a is(typeof()) returned false. It's
quite a rare situation anyway.
I don't know if these semantics are sufficiently better to justify the
implementation difficulty.

But regardless, given we that we have to have a special semantic pass, specifically for mixin declarations and static if, then the beauty of have everything be order-independent is already damaged. So I think we might as well say that the semantic0 "add symbols into scope" pass applies only to static if and mixins, and is run in the order in which the static ifs and mixins appear in the file; but the semantic meaning of declarations themselves is independent of order.

So I would recommend option (1) from my list above: do static if and mixin in order, then when they're finished, do the declarations in any order.

> Don, just so you know, I've been thinking for a while about transitioning from doing the semantic pass in order to doing it completely "on demand". In other words, try to semantic a declaration. In the process, any declarations it depends on are semantic'd if not already, recursively.

I've been wondering for quite some time about how one could leverage the D
module system to properly parallelize D compilation.
(Of course I'm not speaking about the crappy C way of compiling modules
separately with make -j where everything is processed umpteen times)

I think lexing and parsing could be done in parallel without problems?! Semantic analysis is the tough part but one could probably consider the import graph and process all the sinks, i.e. modules that only import modules already processed or no modules, in parallel and then start a new iteration?!

Now I wonder how that would interact with the proposed on-demand semantic analysis.