Hi Guys,

I have been looking into the DMD now to see what I can do about CTFE.
Unfortunately It is a pretty big mess to untangle.
Code responsible for CTFE is in at least 3 files.
[dinterpret.d, ctfeexpr.d, constfold.d]
I was shocked to discover that the PowExpression actually depends on phobos! (depending on the exact codePath it may or may not compile...)
which let to me prematurely stating that it worked at ctfe [http://forum.dlang.org/thread/ukcoibejffinknrbzktv@forum.dlang.org]

My Plan is as follows.

Add a new file for my ctfe-interpreter and update it gradually to take more and more of the cases the code in the files mentioned above was used for.

Do Dataflow analysis on the code that is to be ctfe'd so we can tell beforehand if we need to store state in the ctfe stack or not.

Or baring proper data-flow analysis: RefCouting the variables on the ctfe-stack could also be a solution.

I will post more details as soon as I dive deeper into the code.

On 09 May 2016 19:01, "Stefan Koch via Digitalmars-d-announce" < digitalmars-d-announce@puremagic.com> wrote:
>
> Hi Guys,
>
> I have been looking into the DMD now to see what I can do about CTFE.
> Unfortunately It is a pretty big mess to untangle.
> Code responsible for CTFE is in at least 3 files.
> [dinterpret.d, ctfeexpr.d, constfold.d]
> I was shocked to discover that the PowExpression actually depends on
phobos! (depending on the exact codePath it may or may not compile...)
> which let to me prematurely stating that it worked at ctfe [
http://forum.dlang.org/thread/ukcoibejffinknrbzktv@forum.dlang.org]
>
> My Plan is as follows.
>
> Add a new file for my ctfe-interpreter and update it gradually to take
more and more of the cases the code in the files mentioned above was used for.
>
> Do Dataflow analysis on the code that is to be ctfe'd so we can tell
beforehand if we need to store state in the ctfe stack or not.
>
> Or baring proper data-flow analysis: RefCouting the variables on the
ctfe-stack could also be a solution.
>
> I will post more details as soon as I dive deeper into the code.
>
>

Will be awesome. Particularly if you document the workings of ctfe, might make a great set of articles for a blog.

Hi Stefan,

On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:
> My Plan is as follows.

I think you guys talked about it at the conference, but be sure to coordinate with Timon Gehr. You'll want to steal all the best ideas from the various implementations anyway. ;)

> Do Dataflow analysis on the code that is to be ctfe'd so we can tell beforehand if we need to store state in the ctfe stack or not.
>
> Or baring proper data-flow analysis: RefCouting the variables on the ctfe-stack could also be a solution.

I presume by "store state" you mean persisting objects beyond the bounds of a single CTFE invocation? My first inclination here would simply be to make all allocations from a new arena each time CTFE is entered (can also re-use memory from prior runs for that), do a deep-copy of the result (converting it to full AST nodes, etc.), and then drop the entire arena. But probably you have thought of (and discarded) this already.

 — David

awesome news :-) thanks you

On Monday, 9 May 2016 at 18:19:53 UTC, David Nadlinger wrote:
> Hi Stefan,
>
> On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:
>> My Plan is as follows.
>
> I think you guys talked about it at the conference, but be sure to coordinate with Timon Gehr. You'll want to steal all the best ideas from the various implementations anyway. ;)
>
>> Do Dataflow analysis on the code that is to be ctfe'd so we can tell beforehand if we need to store state in the ctfe stack or not.
>>
>> Or baring proper data-flow analysis: RefCouting the variables on the ctfe-stack could also be a solution.
>
> I presume by "store state" you mean persisting objects beyond the bounds of a single CTFE invocation? My first inclination here would simply be to make all allocations from a new arena each time CTFE is entered (can also re-use memory from prior runs for that), do a deep-copy of the result (converting it to full AST nodes, etc.), and then drop the entire arena. But probably you have thought of (and discarded) this already.
>
>  — David

The current implementation stores persistent state for every ctfe incovation.
While caching nothing. Not even the compiled for of a function body.
Because it cannot relax purity.
Which is why a simple while-loop from 0 to 100_000_00 can crash dmd if executed at ctfe.

Your advice is a good one. I am happy to discuss with others!
There is no need to duplicate mental workload.

On 5/9/2016 9:57 AM, Stefan Koch wrote:
>[...]

The memory consumption problem, at least, can be resolved by using stack temporaries instead of allocating new nodes. This was already done in constfold.d, but not in the rest of the interpreter.

Doing that will (I predict) also double its speed right there.

On 5/9/16, Stefan Koch via Digitalmars-d-announce <digitalmars-d-announce@puremagic.com> wrote:
> I was shocked to discover that the PowExpression actually depends on phobos!

I seem to remember having to implement diagnostics in DMD asking for the user to import std.math. I'm fairly confident it had something to do with power expressions.

Yah, it's a mess. :)

On Monday, 9 May 2016 at 20:24:56 UTC, Walter Bright wrote:
> On 5/9/2016 9:57 AM, Stefan Koch wrote:
>>[...]
>
> The memory consumption problem, at least, can be resolved by using stack temporaries instead of allocating new nodes. This was already done in constfold.d, but not in the rest of the interpreter.
>
> Doing that will (I predict) also double its speed right there.

Thanks, your advice is most helpful and a good first stop-gap.

Still the current state of CTFE is almost not maintainable
 and I would really prefer a clean-slate approach.

SDC benefits extremely from the extra level of indirection, however I do understand that SDC and DMD have diffrent goals regarding compile-speed.

Also I feel that good code has found it's most beautiful shape when it's simplicity makes it inevitable, at least the Ctfe-Mechanism has not reached this point yet, imo.

On Monday, May 09, 2016 13:24:56 Walter Bright via Digitalmars-d-announce wrote:
> On 5/9/2016 9:57 AM, Stefan Koch wrote:
> >[...]
>
> The memory consumption problem, at least, can be resolved by using stack temporaries instead of allocating new nodes. This was already done in constfold.d, but not in the rest of the interpreter.
>
> Doing that will (I predict) also double its speed right there.

Previously, Don stated that he thought that simply making it so that CTFE didn't allocate a new integer every time it mutated it would be a _huge_ speed-up by itself (which presumably is what you're talking about with regards to allocating new nodes). Unfortunately, he got too busy to actually do that work and no one else stepped in to do. But if Stefan is going to step up and improve CTFE, that's fantastic. It's one of D's best features, but it's also one of its most problematic. Fixng that would be huge.

- Jonathan M Davis

On 5/9/2016 2:32 PM, Andrej Mitrovic via Digitalmars-d-announce wrote:
> On 5/9/16, Stefan Koch via Digitalmars-d-announce
> <digitalmars-d-announce@puremagic.com> wrote:
>> I was shocked to discover that the PowExpression actually depends
>> on phobos!
>
> I seem to remember having to implement diagnostics in DMD asking for
> the user to import std.math. I'm fairly confident it had something to
> do with power expressions.
>
> Yah, it's a mess. :)
>

The pow stuff should just be done in dmd without reference to the library.