On Wednesday, 17 July 2013 at 15:26:19 UTC, John Colvin wrote:
> I think I can speak quite safely for the majority of the community when I say that you are only welcome here if you can keep your aggressive and disrespectful comments to yourself.

I think you cannot, as nobody provided you such right and I, in particular, do object, as saying "you may ask but please without obscene language" is not a way how public technical newsgroup should be organized. It is pretty obvious in this case that flow of obscene language will not stop. Also, your politeness does harm in this case rather than good.

On Wednesday, 17 July 2013 at 16:19:46 UTC, Maxim Fomin wrote:
> On Wednesday, 17 July 2013 at 15:26:19 UTC, John Colvin wrote:
>> I think I can speak quite safely for the majority of the community when I say that you are only welcome here if you can keep your aggressive and disrespectful comments to yourself.
>
> I think you cannot, as nobody provided you such right and I, in particular, do object, as saying "you may ask but please without obscene language" is not a way how public technical newsgroup should be organized. It is pretty obvious in this case that flow of obscene language will not stop. Also, your politeness does harm in this case rather than good.

Sorry, I assumed we were all against having people make aggressive attacks on other members?

Perhaps there is a misunderstanding here: When I say "I think I can quite safely speak for the majority" what I mean is "I'm pretty sure people most people here agree with the following". I don't literally mean I'm some sort of elected mouthpiece.

Btw, I have no problem with the obscene language.

On Wednesday, July 17, 2013 16:02:00 JS wrote:
> foreach(n; std.string.split(s, ","))
> {
> // n can't be read at compile time
> }
> 
> using in a ctfe.
> 
> How to loop over a string array in a ctfe?

I would point out that if you're just splitting a string to iterate over it, then you should probably use std.algorithm.splitter, as it avoids allocating a new array.

But the code that you have here appears to work just fine in CTFE for me, so you probably have something else going on in code that you didn't show which is making it so that CTFE isn't working for you.

- Jonathan M Davis

On Wednesday, 17 July 2013 at 18:00:15 UTC, Jonathan M Davis wrote:
> On Wednesday, July 17, 2013 16:02:00 JS wrote:
>> foreach(n; std.string.split(s, ","))
>> {
>> // n can't be read at compile time
>> }
>> 
>> using in a ctfe.
>> 
>> How to loop over a string array in a ctfe?
>
> I would point out that if you're just splitting a string to iterate over it,
> then you should probably use std.algorithm.splitter, as it avoids allocating a
> new array.
>
> But the code that you have here appears to work just fine in CTFE for me, so
> you probably have something else going on in code that you didn't show which
> is making it so that CTFE isn't working for you.
>
> - Jonathan M Davis

I will try that but the error message given hides the real issue. I think the problem is, as demonstrated, simply wrapping a standard function in a template breaks ctfe.

I believe the issue is more so about the semantic analysis of ctfe more than anything...

I can't put together a working example right now(other things to do) but the gist of the matter is:

template strsplit(string n) { enum strsplit = std.string.split(n, ","); }

... inside a ctfe ...

foreach(n; strsplit!(s)) // doesn't work
foreach(n; std.string.split(s, ",")) // works

To me, this is confusing as hell and "n can't be read at compile time" took me a good few hours to realize it was simply using the template.  I'm guessing that it has something to do with the enum "return type" of strsplit but I've tried other various things.

On Thu, Jul 18, 2013 at 01:02:26AM +0200, JS wrote:
[...]
> I can't put together a working example right now(other things to do)
> but the gist of the matter is:
> 
> template strsplit(string n) { enum strsplit = std.string.split(n,
> ","); }
> 
> ... inside a ctfe ...
> 
> foreach(n; strsplit!(s)) // doesn't work
> foreach(n; std.string.split(s, ",")) // works
> 
> To me, this is confusing as hell and "n can't be read at compile time" took me a good few hours to realize it was simply using the template.  I'm guessing that it has something to do with the enum "return type" of strsplit but I've tried other various things.

I think it will help to understand the difference between templates and CTFE. Even though both are processed at compile-time, they are not the same thing, and mixing them may not always do what you think it's doing.

Basically, a template is used for generating code, so any template parameters must be known "at compile-time", which means that it must ultimately reduce to a typename, or a literal of a built-in type. Note that this is NOT the same thing as "this variable is being used by CTFE so the compiler should know what its value is at `compile-time'". At this conceptual stage of compilation, the compiler hasn't fully transformed the source code into runnable form yet, so the only thing it can do at this point is to work with, conceptually-speaking, text-level entities. It can't handle variables and other such things that only exist when a program has been transformed into a runnable form and executed by an interpreter.

CTFE is basically a subsystem of the compiler that has the ability to run code *after* it has been generated (either via parsing the source code directly, or via expanding templates). That is to say, the program is now in a (semi)runnable form, but the compiler hasn't output the object files / executables yet. Basically kinda like an "early execution" of your program, if you will. Since this happens in a conceptually later stage than template expansion, whatever's going on in CTFE generally can't be fed back into template parameters. Template expansion must have been completed before CTFE even comes into play.

I think part of the confusion may stem from the liberal use of the term "compile-time", which may have given a false impression that if entity X is known "at compile-time", then it should be usable by construct Y which requires stuff that's known "at compile-time". This oversimplifies the process of compilation; in reality, "compile-time" consists of several distinct conceptual phases (in implementation this may not necessarily be true literally, but it helps to understand the conceptual stages). What generally happens is that the compiler needs to:

1) Read the text representation of the source code, break that down
into tokens ("lexing"), and based on the sequence of tokens, construct
an abstract syntax tree that represents the structure of the source code
("parsing");

2) Translate the syntax tree into an internal representation (IR) of the program that more-or-less maps to the machine code that will eventually be output;

3) Translate the IR into actual instructions the target CPU can understand, and output that as the object file / executable.

Conceptually-speaking, template expansion comes somewhere between (1) and (2): it's a mechanism for producing large numbers of very-similar subtrees of the syntax tree without requiring the programmer to do lots and lots of typing. CTFE is roughly somewhere between (2) and (3): the program isn't fully compiled yet, but enough translation has been done that it is now in a form that can actually be run (by an interpreter). In order to get to this point, template expansion must have been completed first, since otherwise your syntax tree isn't complete yet, so by definition it can't have been translated into a runnable form yet. But since template expansion must have already taken place, that means you can't feed CTFE values back into templates -- since otherwise template expansion couldn't have already taken place!

So, long story short, template parameters must be known at "text-level", if we can coin a term for that, the compiler must be able to reduce said parameters into a concrete type name or a literal of a built-in type. CTFE, on the other hand, works after template expansion has completed (conceptually speaking), and so is essentially a kind of "early execution" of your program. CTFE variables and the like are "interpreter-level", so they can't be passed back into template parameters that expect "text-level" input.

The actual details of what the compiler does is, of course, quite a bit more complex than the above (over)simplified description, but generally speaking, if you stick by the rule that CTFE values can't be fed back into the templating system, you will experience much less frustration.


T

-- 
There are two ways to write error-free programs; only the third one works.

On Wednesday, 17 July 2013 at 23:56:11 UTC, H. S. Teoh wrote:
> On Thu, Jul 18, 2013 at 01:02:26AM +0200, JS wrote:
> [...]
>> I can't put together a working example right now(other things to do)
>> but the gist of the matter is:
>> 
>> template strsplit(string n) { enum strsplit = std.string.split(n,
>> ","); }
>> 
>> ... inside a ctfe ...
>> 
>> foreach(n; strsplit!(s)) // doesn't work
>> foreach(n; std.string.split(s, ",")) // works
>> 
>> To me, this is confusing as hell and "n can't be read at compile
>> time" took me a good few hours to realize it was simply using the
>> template.  I'm guessing that it has something to do with the enum
>> "return type" of strsplit but I've tried other various things.
>
> I think it will help to understand the difference between templates and
> CTFE. Even though both are processed at compile-time, they are not the
> same thing, and mixing them may not always do what you think it's doing.
>
> Basically, a template is used for generating code, so any template
> parameters must be known "at compile-time", which means that it must
> ultimately reduce to a typename, or a literal of a built-in type. Note
> that this is NOT the same thing as "this variable is being used by CTFE
> so the compiler should know what its value is at `compile-time'". At
> this conceptual stage of compilation, the compiler hasn't fully
> transformed the source code into runnable form yet, so the only thing it
> can do at this point is to work with, conceptually-speaking, text-level
> entities. It can't handle variables and other such things that only
> exist when a program has been transformed into a runnable form and
> executed by an interpreter.
>
> CTFE is basically a subsystem of the compiler that has the ability to
> run code *after* it has been generated (either via parsing the source
> code directly, or via expanding templates). That is to say, the program
> is now in a (semi)runnable form, but the compiler hasn't output the
> object files / executables yet. Basically kinda like an "early
> execution" of your program, if you will. Since this happens in a
> conceptually later stage than template expansion, whatever's going on in
> CTFE generally can't be fed back into template parameters. Template
> expansion must have been completed before CTFE even comes into play.
>
> I think part of the confusion may stem from the liberal use of the term
> "compile-time", which may have given a false impression that if entity X
> is known "at compile-time", then it should be usable by construct Y
> which requires stuff that's known "at compile-time". This oversimplifies
> the process of compilation; in reality, "compile-time" consists of
> several distinct conceptual phases (in implementation this may not
> necessarily be true literally, but it helps to understand the conceptual
> stages). What generally happens is that the compiler needs to:
>
> 1) Read the text representation of the source code, break that down
> into tokens ("lexing"), and based on the sequence of tokens, construct
> an abstract syntax tree that represents the structure of the source code
> ("parsing");
>
> 2) Translate the syntax tree into an internal representation (IR) of the
> program that more-or-less maps to the machine code that will eventually
> be output;
>
> 3) Translate the IR into actual instructions the target CPU can
> understand, and output that as the object file / executable.
>
> Conceptually-speaking, template expansion comes somewhere between (1)
> and (2): it's a mechanism for producing large numbers of very-similar
> subtrees of the syntax tree without requiring the programmer to do lots
> and lots of typing. CTFE is roughly somewhere between (2) and (3): the
> program isn't fully compiled yet, but enough translation has been done
> that it is now in a form that can actually be run (by an interpreter).
> In order to get to this point, template expansion must have been
> completed first, since otherwise your syntax tree isn't complete yet, so
> by definition it can't have been translated into a runnable form yet.
> But since template expansion must have already taken place, that means
> you can't feed CTFE values back into templates -- since otherwise
> template expansion couldn't have already taken place!
>
> So, long story short, template parameters must be known at "text-level",
> if we can coin a term for that, the compiler must be able to reduce said
> parameters into a concrete type name or a literal of a built-in type.
> CTFE, on the other hand, works after template expansion has completed
> (conceptually speaking), and so is essentially a kind of "early
> execution" of your program. CTFE variables and the like are
> "interpreter-level", so they can't be passed back into template
> parameters that expect "text-level" input.
>
> The actual details of what the compiler does is, of course, quite a bit
> more complex than the above (over)simplified description, but generally
> speaking, if you stick by the rule that CTFE values can't be fed back
> into the templating system, you will experience much less frustration.
>
>
> T


Thanks, this has made it much clearer.

Something like

foreach(a; StrSplit!(s))
   foreach(b; StrSplit !(a))

does work because the second StrSplit uses a "ctfe-time variable" instead of a "template-time variable".

My logic was:

1. first StrSplit resolved
2. first foreach evaluated
3. second StrSplit resolved
4. second foreach evaluated

while it actually is

1. first StrSplit resolved
2. second StrSplit resolved
3. foreach's resolved

because template expansion happens before any ctfe expansion.

I guess I was thinking the compiler would be smart enough to interleave template expansion and ctfe code(which would be much more powerfull).

Effectively template expansion is a sort of pre-processing to ctfe code and must be static as far as ctfe's go.

Does this make sense?

On Thu, Jul 18, 2013 at 07:23:57AM +0200, JS wrote:
[...]
> Thanks, this has made it much clearer.
> 
> Something like
> 
> foreach(a; StrSplit!(s))
>    foreach(b; StrSplit !(a))
> 
> does work because the second StrSplit uses a "ctfe-time variable" instead of a "template-time variable".
> 
> My logic was:
> 
> 1. first StrSplit resolved
> 2. first foreach evaluated
> 3. second StrSplit resolved
> 4. second foreach evaluated
> 
> while it actually is
> 
> 1. first StrSplit resolved
> 2. second StrSplit resolved
> 3. foreach's resolved
> 
> because template expansion happens before any ctfe expansion.
> 
> I guess I was thinking the compiler would be smart enough to interleave template expansion and ctfe code(which would be much more powerfull).
> 
> Effectively template expansion is a sort of pre-processing to ctfe code and must be static as far as ctfe's go.
[...]

That's one way to think of it, yes.

As for interleaving template expansion vs. ctfe evaluation, the compiler *does* do that to some extent.  For example, this code does work:

	// Function that can be evaluated by CTFE
	int func(int x) pure {
		int sum;
		foreach (i; 0..x) {
			sum += i;
		}
		return sum;
	}

	// Force CTFE evaluation
	enum myConst = func(10);

	// Template that requires an int parameter.
	template MyTemplate(int x) {
		enum MyTemplate = x+10;
	}

	// Instantiate template with enum produced by CTFE.
	pragma(msg, MyTemplate!myConst);

	void main() {}

The reason this works is because the compiler is smart enough to figure out that myConst requires func, so it first compiles func far enough to be CTFE-evaluable, then it evaluates func to produce the value of myConst, and then myConst is used to instantiate MyTemplate. You could think of it as the compiler compiling different parts of the program at different rates, so func has been compiled into a runnable state, but the pragma(msg) line is still at the template expansion state.

Note, however, that the template-before-CTFE limitation still applies: func can't require template expansion while it's running; it must be entirely compilable into runnable state before CTFE can evaluate it. Other parts of the program can still remain at the template-expansion stage, so they can take some CTFE-produced values as template parameters. But you can't make any reverse dependencies / loops.  You're OK if you already have runnable code that can then produce template parameters for other code, but you can't run CTFE and template expansion simultaneously in the *same* code.

So the compiler is actually pretty smart about reordering these things, but the fundamental limitation of template-before-CTFE still applies to each individual code unit. After all, you can't run code that hasn't been fully expanded by the template system yet.


T

-- 
Winners never quit, quitters never win. But those who never quit AND never win are idiots.

On Thursday, 18 July 2013 at 05:57:48 UTC, H. S. Teoh wrote:
> On Thu, Jul 18, 2013 at 07:23:57AM +0200, JS wrote:
> [...]
>> Thanks, this has made it much clearer.
>> 
>> Something like
>> 
>> foreach(a; StrSplit!(s))
>>    foreach(b; StrSplit !(a))
>> 
>> does work because the second StrSplit uses a "ctfe-time variable"
>> instead of a "template-time variable".
>> 
>> My logic was:
>> 
>> 1. first StrSplit resolved
>> 2. first foreach evaluated
>> 3. second StrSplit resolved
>> 4. second foreach evaluated
>> 
>> while it actually is
>> 
>> 1. first StrSplit resolved
>> 2. second StrSplit resolved
>> 3. foreach's resolved
>> 
>> because template expansion happens before any ctfe expansion.
>> 
>> I guess I was thinking the compiler would be smart enough to
>> interleave template expansion and ctfe code(which would be much more
>> powerfull).
>> 
>> Effectively template expansion is a sort of pre-processing to ctfe
>> code and must be static as far as ctfe's go.
> [...]
>
> That's one way to think of it, yes.
>
> As for interleaving template expansion vs. ctfe evaluation, the compiler
> *does* do that to some extent.  For example, this code does work:
>
> 	// Function that can be evaluated by CTFE
> 	int func(int x) pure {
> 		int sum;
> 		foreach (i; 0..x) {
> 			sum += i;
> 		}
> 		return sum;
> 	}
>
> 	// Force CTFE evaluation
> 	enum myConst = func(10);
>
> 	// Template that requires an int parameter.
> 	template MyTemplate(int x) {
> 		enum MyTemplate = x+10;
> 	}
>
> 	// Instantiate template with enum produced by CTFE.
> 	pragma(msg, MyTemplate!myConst);
>
> 	void main() {}
>
> The reason this works is because the compiler is smart enough to figure
> out that myConst requires func, so it first compiles func far enough to
> be CTFE-evaluable, then it evaluates func to produce the value of
> myConst, and then myConst is used to instantiate MyTemplate. You could
> think of it as the compiler compiling different parts of the program at
> different rates, so func has been compiled into a runnable state, but
> the pragma(msg) line is still at the template expansion state.
>
> Note, however, that the template-before-CTFE limitation still applies:
> func can't require template expansion while it's running; it must be
> entirely compilable into runnable state before CTFE can evaluate it.
> Other parts of the program can still remain at the template-expansion
> stage, so they can take some CTFE-produced values as template
> parameters. But you can't make any reverse dependencies / loops.  You're
> OK if you already have runnable code that can then produce template
> parameters for other code, but you can't run CTFE and template expansion
> simultaneously in the *same* code.
>
> So the compiler is actually pretty smart about reordering these things,
> but the fundamental limitation of template-before-CTFE still applies to
> each individual code unit. After all, you can't run code that hasn't
> been fully expanded by the template system yet.
>
>
> T

Thanks. Your reply was extremely helpful and I think I'll have less pms over ctfe's.

Since you do such a good job a explaining things, would you mind informing me what the difference between alias T, T and string T used as template parameters are?

template A!(T) { }
template A!(alias T) { }
template A!(string T) { }


The string version is suppose to be a specialization of A that is called when A is passed a string? e.g., A!("...") calls the string version? (this is my assumption but I seem to run into problems where sometimes other versions are called)

The alias T version seems to accept a symbol alias(a sort of redirection/substitution is made but no actual evaluation is made on the symbol).

The T version accepts types and compile type constants/literals(like a string).


This is what I understand them to do... but always get into trouble where something wants an alias while another thing wants a type.

moduleName complains a lot because sometimes I pass it a built in type like double , sometimes a string(by accident but I want it to return the string itself), and sometimes I pass a user type.

I have to rectify all 3 usages into one common template so I wrap moduleName into, say ModuleName.

template ModuleName(T) { return (isBasicType!T) ? "" : moduleName!(T); }
template ModuleName(string s) { return s; }

but sometimes, it seems the T version is called when I pass it a string.

Error: template instance moduleName!(string) does not match template declaration moduleName(alias T)

when I use

template ModuleName(T)
{
    static if (isValueType!T)
        enum ModuleName = "";
    else static if (isString!T)
        enum ModuleName = T;
    else
        enum ModuleName = StripStr!(std.traits.moduleName!T, `"`, " ");
}

template ModuleName(string s) { enum ModuleName = s; }

StripStr is just a template that strips the ends of a string. isValueType is a template wrapper to test T for a value type(scalar, void, string... *all* types that do not have/need module resolution)

The error suggests that I'm passing a string into the std.traits.moduleName call but this shouldn't happen?

Ok, I see, I am actually passing it a string type(not a string literal) to the template so the first is chosen. I guess my isValuetype isn't working properly ;/