March 17, 2017
On Friday, 17 March 2017 at 14:27:25 UTC, Adam D. Ruppe wrote:
> On Friday, 17 March 2017 at 13:53:58 UTC, Hussien wrote:
>> Yes, but you have a nested foreach loop. One runtime and one compile time. The break goes with the runtime loop... but NORMAL programming logic tells us that the break goes with the loop that it exists in.
>
> It did. It broke the loop counting the numbers, but continued the loop of strings normally.
>
>
> People have requested a `static foreach` before, but that request has not yet been granted.
>
>> They are going to expect them to continue or break out of the inner most loop that they are called from.
>
> That is exactly what they do, and exactly what happened in all the examples posted.

You're failing to see what I am talking about. I understand the compiled code is correct and functions as it should.

What I am talking about is

how the statement

static if (x) continue;
pragma(msg, "called");

vs

static if (x) { } else
pragma(msg, "not called");


They are both semantically equivalent

e.g.,

if (x) continue; foo();

vs

if (x) {} else { foo(); }

Both do the same thing in loops when used in the context I originally gave. e.g., using a continue to skip over elements of an interation. Works great in normally programming!

BUT in static programming, the continue FUNCTIONS as a runtime component... just as you say. BUT that is not how one interprets it on the static level.

The logic for


if (x) continue; foo();

vs

if (x) {} else { foo(); }

and the logic for


static if (x) continue; foo();

vs

static if (x) {} else { foo(); }


should be identical. Just because we are programming on two different levels doesn't change the logic(or shouldn't).

The problem, of course, is the continue does exactly what you claim it does.

But when I am meta programming I am thinking in terms of meta programming. I program exactly the same way as I do in runtime programming except for the obvious shifts that are required. But foreach is a foreach. In runtime it works on runtime stuff and at compile time it works on compile time stuff.

Effectively I have

runtime_foreach and static_foreach in my head and I magically disambiguate when required.

But there is no such concept for continue. It is impossible to "continue" or even "break" a static for each loop. That is what I am addressing.

SO, because I had a mental glitch and I used continue "thinking" it would work for the static case, e.g., I assumed it was a static_continue... when no such thing exists, the code compiled and did something I did not expect!

YES, It was my fault! BUT the point is that D has that ambiguity. AND because the flow structures for runtime and compile time are virtually identical, the mistake is easy to make! Even if not, if the mistake does occur, it will be a difficult thing to track down in real world cases that are complex. That is the only point I am making! It has nothing to do with what D actually does, but how it does it.

A continue or break should create a warning when used in the context of a static foreach. OR change the freaken syntax of static semantics vs runtime. Making them identical is ridiculously and only leads to these kinds of problems. Sure, it works in 99% of the cases, but so will a new syntax.

My point is not what D does specifically in compiling source code, or that it does it right or wrong. I am talking about the interpretation by a human of the code. $@#$@($*%**%*#(@(@!)@$_@%%#$#$@#@($!* is some code to some yet to be created programming language... but it doesn't mean that humans will be able to interpret it properly. I am talking about how humans interpret things.

Do you really think that a non-D programmer that sees the code

foreach(x; [1,2,3])
foreach(y; aliasSeqOf![1,2,3])
static if (y == 1) continue;
pragma(msg, y)

is going to be able to interpret the code correctly?

Because if they use a rewrite rule, which is GENERALLY VALID in all programming languages:

foreach(x; [1,2,3])
foreach(y; aliasSeqOf![1,2,3])
static if (y == 1) {} else
{
pragma(msg, y)
}

(again, this should be the same as the first case, because we can do that at runtime and it is valid... people write code like that)

Yet one gets a different result.

Hence, if they are programming some NASA rocket, and make that mistake and the rocket goes left instead of dropping it's boosters, then what? Who's fault? Is it D's fault for creating ambiguity between static and runtime programming? Of course it is, because It sets the foundation for what is to come.

Again, my point is not about D compiling the above code, but about the syntax/grammar/language issue it creates conceptually. You may not have that problem because you are familiar enough with D's meta programming but to someone that isn't, they have to go on what they know, and there is a problem conceptually they will have to learn about. It shouldn't be that way. It is a problem of D treating apples as oranges. foreach != foreach... does that make sense? Sure, if you already learned, but if you are learning or have a brain fart, too bad for you...

This is D's approach: "Meta programming - The same as programming... well almost". The almost part is what gets people killed.














March 17, 2017
On Friday, 17 March 2017 at 15:14:08 UTC, Hussien wrote:
> What I am talking about is

If you want to add a new feature, `static foreach`, that has static continue and static break, I can get behind that, but that's a new feature, not a bug in the existing feature. You think it is something it isn't.

`static if` is unique in D. Nothing else works like it. There is no static switch, no static foreach. Regular switch and foreach can work on compile time values, but they don't actually change their behavior like static if does, they work the same as always. The best you get is that the current value of the foreach can inherit the compile-time staticness of what it is looping, allowing you to use it in a few more places (notably in static if), but the loop construct itself is the same.

That's it. It does not do special flow control, it does not change scopes or compile requirements.

A new `static foreach` that DOES do those things, and works in places `static if` currently works like outside functions, would be a cool new feature. But until that's added, stop thinking of "static foreach" as being a special thing in the language now. It isn't there. It is just regular foreach and thinking of it as that, always, will make things a lot easier.


Then you can join the calls for a new `static foreach` that explicitly IS compile time! And we'll be on the same page there.
March 17, 2017
On Fri, Mar 17, 2017 at 03:14:08PM +0000, Hussien via Digitalmars-d-learn wrote: [...]
> What I am talking about is
> 
> how the statement
> 
> static if (x) continue;
> pragma(msg, "called");
> 
> vs
> 
> static if (x) { } else
> pragma(msg, "not called");
> 
> 
> They are both semantically equivalent

This appears to be yet another case of the term "compile-time" causing confusion, because it's actually an ambiguous term.  I actually think it's a bad term that should be replaced by something more precise; its ambiguity has caused confusion on the part of D learners more often than not.

There are actually (at least) TWO distinct phases of compilation that are conventionally labelled "compile time":

1) Template expansion / AST manipulation, and:

2) CTFE (compile-time function evaluation).

Not clearly understanding the distinction between the two often leads to confusion and frustration at why the compiler isn't doing "what I want".

This confusion is confounded by the fact that these two do mutually interact in any non-trivial metaprogramming D code, often in rather complex ways. But the fundamental thing to understand here is:

	Template expansion / AST manipulation must be completed *before*
	CTFE can run.

Basically, CTFE is essentially a D interpreter embedded inside the compiler, that simulates computation at runtime.  As such, it requires the AST to be fully compiled, as in, the compiler is ready to generate object code for it, before CTFE can work.  This is because it makes no sense to generate code on an incomplete / partial AST.

Furthermore, once a piece of code has made it to the CTFE stage, its AST has already been processed, and it's now compiled into an internal representation (analogous to bytecode), so AST-manipulating constructs no longer make any sense.  In the CTFE stage, there is no such thing as an AST anymore.

Constructs like `static if` and `pragma(msg)` belong to the AST manipulation stage. `static if` changes the effective AST that the compiler sees when it's generating code, and pragma(msg) is essentially a debugging construct that says "print this message if this part of the AST makes it past the template expansion phase". This is why code like this one doesn't do what you might think it does:

	int ctfeFunc(bool b) {
		if (b) {
			pragma(msg, "hello");
			return 1;
		} else {
			pragma(msg, "goodbye");
			return 2;
		}
	}

The compiler will print *both* "hello" and "goodbye", because the pragma(msg) is evaluated *when the AST of this function is processed*. The variable 'b' doesn't even exist at that point, because ASTs don't have the concept of a 'variable' -- 'b' is just an identifier node in the tree. And 'if' is just another node in the tree that represents a control-flow directive.  Change that to 'static if', however, and the compiler sees it differently:

	int ctfeFunc(bool b) {
		static if (b) {
			pragma(msg, "hello");
			return 1;
		} else {
			pragma(msg, "goodbye");
			return 2;
		}
	}

This code won't work, because 'b' is a variable, but variables don't exist at the AST manipulation stage. So you have to turn 'b' into a so-called "compile-time argument" (boy I hate that term "compile-time", it totally is not precise enough to convey the meaning here):

	int ctfeFunc(bool b)() {
		static if (b) {
			pragma(msg, "hello");
			return 1;
		} else {
			pragma(msg, "goodbye");
			return 2;
		}
	}

Do not be deceived by the appearances; the story here is far more involved than "since static if is a `compile-time` construct, obviously it needs the condition to be made from `compile-time` arguments". You have to understand that when you move 'b' into the so-called compile-time parameter list, you're essentially saying "here's a template that, given a boolean value, produces an AST tree according to the following pattern".  If you instantiate ctfeFunc!true, then what effectively happens is that the compiler sees this AST:

	int ctfeFunc!true() {
		return 1;
	}

So it prints "hello". Note that the else branch is NOT EVEN SEEN past the AST stage. It practically doesn't exist at that point. The pragma(msg) is also not seen past that point: it is consumed in the AST manipulation stage. The compiler sees the static if, evaluates the condition, and basically prunes the else branch off, then sees the pragma(msg) and emits the message (i.e., acknowledging "yes this part of the AST makes it past the AST manipulation stage"). The pragma(msg) is pruned from the AST after that.

By the time CTFE runs on this function, all the CTFE interpreter sees is "return 1;".  It doesn't see the "static if" nor the "pragma(msg)", because those things are AST manipulation constructs; past the AST stage such things don't exist anymore.


What complicates this 2-stage picture, though, and probably what doesn't help the confusion with the term "compile-time", is that the compiler is smart enough to perform CTFE on-demand. Meaning that it's valid to do this:

	int ctfeFunc(bool b) { /* N.B.: "runtime" parameter! */
		if (b) return 1;
		else return 2;
	}

	enum b = true;
	static if (ctfeFunc(b) == 1)
		struct S { int x; }
	else
		struct S { int y; }

Note that the static if here has a condition that depends on the output of a CTFE function. This appears be a reversal of the AST manipulation / CTFE stages, but strictly speaking that's actually not the case.  What actually happens is:

1) The compiler sees the declaration of ctfeFunc(), produces an AST for it.
   Note that this means the AST of ctfeFunc gets finalised here -- any
   static if's, templates, etc., are expanded into the "final" AST for
   this function.

2) Then the compiler sees the static if, and enters the AST phase *for
   this part of the code*. It says, I need to know the value of
   ctfeFunc(b) in order to know which struct declaration to use -- so it
   *emits code* for ctfeFunc(), and then runs the CTFE engine on the
   resulting code.  Then it takes the output of that evaluation to make
   the decision.

The key point here is that ctfeFunc has *already* passed the AST manipulation stage while the struct declaration is still in the AST stage.  The compiler is clever enough to sequence the processing of these two parts of the code so that it's able to compute the static if condition.  But the principle of AST manipulation coming before CTFE still applies -- it's not possible for the CTFE engine to interpret ctfeFunc if its AST is still being manipulated (because if the AST is not finalized yet, there isn't any code to interpret!), and it's not possible for the AST manipulation stage to read the value of a CTFE computation that's being run on the same piece of code (because CTFE can't run in the first place while the AST is still being manipulated, and if the code is already ready for CTFE to interpret, that means the AST has already been finalized, you can't change it anymore).

Coming back to your loop:

	static if (x) continue;
	foo();

In the AST manipulation stage, the compiler evaluates x, and if x is true, then the effective AST is:

	continue;
	foo();

By the time it gets to CTFE, it has already "forgotten" that there was such a thing as a static if in the source code. So it will interpret the "continue" unconditionally.

Whereas if you wrote:

	static if (x) {} else { foo(); }

in the AST manipulation stage, if x is true, then the effective AST is:

	{}

and if x is false, then the effective AST is:

	foo();

which is what you intended.


T

-- 
A mathematician is a device for turning coffee into theorems. -- P. Erdos
March 17, 2017
On 03/17/2017 12:05 PM, H. S. Teoh via Digitalmars-d-learn wrote:

> 1) Template expansion / AST manipulation, and:
>
> 2) CTFE (compile-time function evaluation).
>
> Not clearly understanding the distinction between the two often leads to
> confusion and frustration at why the compiler isn't doing "what I want".

I always find your posts very informative and very well articulated. Where is the book already? :) Or blog posts?

Ali

March 17, 2017
On Friday, 17 March 2017 at 19:05:20 UTC, H. S. Teoh wrote:
> On Fri, Mar 17, 2017 at 03:14:08PM +0000, Hussien via Digitalmars-d-learn wrote: [...]
>> [...]
>
> This appears to be yet another case of the term "compile-time" causing confusion, because it's actually an ambiguous term.  I actually think it's a bad term that should be replaced by something more precise; its ambiguity has caused confusion on the part of D learners more often than not.
>
> [...]

Thanks for agreeing with me ;)
March 18, 2017
On Fri, Mar 17, 2017 at 02:52:39PM -0700, Ali Çehreli via Digitalmars-d-learn wrote:
> On 03/17/2017 12:05 PM, H. S. Teoh via Digitalmars-d-learn wrote:
> 
> > 1) Template expansion / AST manipulation, and:
> > 
> > 2) CTFE (compile-time function evaluation).
> > 
> > Not clearly understanding the distinction between the two often leads to confusion and frustration at why the compiler isn't doing "what I want".
> 
> I always find your posts very informative and very well articulated. Where is the book already? :) Or blog posts?
[...]

Haha, I don't think I'm up for writing a book... and I don't really keep a blog either.  But perhaps a writeup on wiki.dlang.org is in order.

This particular topic, I think, is something somebody *should* write about, because it seems to trip up newbies quite often.  Maybe sometimes even seasoned D users get tripped up, too (though they tend to know better where the problem is and how to work around it).  And this isn't the first time I wrote about it.  I guess it's time to consolidate what I've written into a wiki article.


T

-- 
Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan
March 18, 2017
On Saturday, March 18, 2017 17:18:15 H. S. Teoh via Digitalmars-d-learn wrote:
> On Fri, Mar 17, 2017 at 02:52:39PM -0700, Ali Çehreli via Digitalmars-d-
learn wrote:
> > On 03/17/2017 12:05 PM, H. S. Teoh via Digitalmars-d-learn wrote:
> > > 1) Template expansion / AST manipulation, and:
> > >
> > > 2) CTFE (compile-time function evaluation).
> > >
> > > Not clearly understanding the distinction between the two often leads to confusion and frustration at why the compiler isn't doing "what I want".
> >
> > I always find your posts very informative and very well articulated. Where is the book already? :) Or blog posts?
>
> [...]
>
> Haha, I don't think I'm up for writing a book... and I don't really keep a blog either.  But perhaps a writeup on wiki.dlang.org is in order.
>
> This particular topic, I think, is something somebody *should* write about, because it seems to trip up newbies quite often.  Maybe sometimes even seasoned D users get tripped up, too (though they tend to know better where the problem is and how to work around it).  And this isn't the first time I wrote about it.  I guess it's time to consolidate what I've written into a wiki article.

For the most part, I understand compile-time vs runtime quite well - in particular, I don't find templates vs CTFE to be all that complicated once you get into it. I find it to usually be straightforward and obvious, but there are corner cases where it gets harder to understand, and in particular, foreach with compile-time constructs gets weird enough that - to me at least - it stops being straightforward. So, to a great extent, with foreach and compile-time constructs, my understanding is primarily limited to what I need to work in order to do the stuff that I do. And I'm obviously not a D newbie. For newbies, the whole thing is going to be far harder to understand, and for whatever reason, the fact that a function run with CTFE can't use the template parameters beyond what can be done at runtime throws off a lot of people.

- Jonathan M Davis


March 19, 2017
On Sunday, 19 March 2017 at 00:18:15 UTC, H. S. Teoh wrote:

>
> Haha, I don't think I'm up for writing a book... and I don't really keep a blog either.  But perhaps a writeup on wiki.dlang.org is in order.
>
> This particular topic, I think, is something somebody *should* write about, because it seems to trip up newbies quite often.  Maybe sometimes even seasoned D users get tripped up, too (though they tend to know better where the problem is and how to work around it).  And this isn't the first time I wrote about it.  I guess it's time to consolidate what I've written into a wiki article.
>

Ahem. https://dlang.org/blog/

I'm always ready to talk about guest posts at aldacron@gmail.com :-)


March 21, 2017
On Friday, 17 March 2017 at 19:05:20 UTC, H. S. Teoh wrote:
>
> There are actually (at least) TWO distinct phases of compilation that are conventionally labelled "compile time":
>
> 1) Template expansion / AST manipulation, and:
>
> 2) CTFE (compile-time function evaluation).
>
> [ ... ]
> 	Template expansion / AST manipulation must be completed *before*
> 	CTFE can run.
Only the templates that the ctfe relies on.
> [ ... ]
> This is because it makes no sense to generate code on an incomplete / partial AST.

This is not exactly true whenever you use ctfe to generate a source string that you later mix-in. You are working with a partial ast.
>
> Furthermore, once a piece of code has made it to the CTFE stage, its AST has already been processed, and it's now compiled into an internal representation (analogous to bytecode), so AST-manipulating constructs no longer make any sense.

Yes.

> In the CTFE stage, there is no such thing as an AST anymore.
There is an AST. It's just already processed.


March 22, 2017
On Friday, 17 March 2017 at 19:05:20 UTC, H. S. Teoh wrote:
> There are actually (at least) TWO distinct phases of compilation that are conventionally labelled "compile time":
>
> 1) Template expansion / AST manipulation, and:
>
> 2) CTFE (compile-time function evaluation).

This was an awesome explanation, please do put it on the wiki or as a pull request for the website.