On Friday, 31 May 2019 at 07:03:56 UTC, Seb wrote:
>
> FWIW we have been doing this for years now (have a look at the various templates in object.d, e.g. __cmp or __switch) and the fact that no one noticed nor complained, shows that this is no concern.
>
> tl;dr: the compiler already generates a special template everytime you do e.g. a switch, a cast, a comparison between classes etc.
>
> Examples:
> - https://run.dlang.io/is/gsCSO1
> - https://run.dlang.io/is/ti7rlu
>
> There are many more.

Thanks. That clears things up.

Am Fri, 31 May 2019 07:03:56 +0000 schrieb Seb:

> 
> FWIW we have been doing this for years now (have a look at the various templates in object.d, e.g. __cmp or __switch) and the fact that no one noticed nor complained, shows that this is no concern.
> 
> tl;dr: the compiler already generates a special template everytime you do e.g. a switch, a cast, a comparison between classes etc.
> 
> Examples:
> - https://run.dlang.io/is/gsCSO1 - https://run.dlang.io/is/ti7rlu
> 
> There are many more.

In addition, I don't think these hooks will be 'heavy', recursive templates or something similar. Most of them should be quite simple.

Also, if we implement containers in phobos we also use templates. So there's no difference in implementing a templated array manually or making the compiler use templated hooks. In the end, it will have the same overhead (Though builtin slices are of course very commonly used).

-- 
Johannes

On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
> [...]

I forgot to post the updates after week one due to school and
development, so here is my updates for week one and two.


Week 1
======

This week
~~~~~~~~~

- Turned `_d_arrayappendT` and `_d_arrayappendcTX`, with their `*Trace`
  function into templates.
  - `_d_arrayappendcd` and `_d_arrayappendcd` where left alone as they
    are wrapper around the `_d_arrayappendT` for when assigning a dchar
    to char[], and dchar to wchar[], respectively.
- Started working on the dmd code.
  - Got the lowering code working.
  - Currently battling with the CTFE.
- I started with the templates inside object.d but then moved them to
  rt/lifetime.d and did a `public import` inside object.d instead. I did
  this because I started getting template recursions problems when I use
  `externDFunc`, as it used string concatenations internally. I can
  probably later move it back to object.d if needed later when I get the
  CTFE code working again.


Next week
~~~~~~~~~

- Continue with the dmd patches and get it be able to compile.
- Submit both the druntime and the dmd PR for review.


Blockers
~~~~~~~~

In the beginning I tried to translate the hook and all of the functions
it called to templates. This just got me into a wormhole of needed
changes, and it would have made the PRs harder to review. So I restarted
from scratch and instead only translated the entrypoints of each hook,
and then internally use `typeid(T)` to pass to the function
underneath. Future work when all hooks are translated, could be to
translate the functions underneath, when applicable.

When I got the hook lowering working I realized how much the hooks have
lied about their safety. So I had to add a `@trusted` wrapper that fakes
`pure nothrow` for the hook to not break all code. [1]

As mentioned above, I'm currently battling with the CTFE to understand
my lowered hooks. I found examples of this is done for `__ArrayPostblit`
and `__ArrayDtor` inside dinterpret.d, and I will try and understand how
they do it and implement the same for my hooks.


Week 2
======

This week
~~~~~~~~~

- I continued to work on the code to a state were I felt I could start
  to get some feedback, even if the code was not in a working state.
- I got the CTFE 'intercept' code working so the hooks are now using the
  dinterpret.d code instead.
- Submitted work-in-progress PRs
  - druntime: [https://github.com/dlang/druntime/pull/2632]
  - dmd: [https://github.com/dlang/dmd/pull/9982]


Next week
~~~~~~~~~

- I still need to get the PR into a working and mergeable state, I will
  try and get this done as soon as possible.
- I need to implement unittests for the dmd patches to verify that the
  lowering code works as expected, both for the non-trace and trace
  versions.
- When waiting for reviews I will work on the conversion of
  _d_array{setctor,ctor,cat*}, as according to my proposal document.

- I will try to get a PR of this submitted earlier than what I did the
  previous ones. This should be possible as these functions are simpler
  than the last ones and now I know that I should not try and translate
  dependencies and instead just add `typeid(T)` whenever necessary.


Blockers
~~~~~~~~

I've had compilation problem related to the translation of
_d_arrayappendcTX. I fixed this by instead calling the TypeInfo version
of that function.

Due to the exam and school I was not able to finish what I allocated
according to my proposal.

Currently I just have small bug where I trigger the postblit one too
many times with assigning a single element `arr ~= structVar`. This is
because the old backend code assigned the element outside of the
_d_arrayappend call, verses me now doing it internally. I probably just
need to need to make sure that it always lowerest the `structVar` into a
ref-able form, or make it assign the variable outside the hook again.



Footnotes
======
[1]
[https://github.com/Vild/druntime/blob/8f72a8b25f02653445e1b8288583788711b87595/src/rt/lifetime.d#L2871-L2876]

On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
> [...]


Week 3
======

This week
~~~~~~~~~

- Debugged the phobos unittest problems for the _d_arrayappendcT PRs.
  - The problem is probably that the order the postblit and the
    assignments are swapped compared to the current hook
    implementation. The assignment and the postblit probably need to be
    moved to the outside of the to _d_arrayappendcT function.
- Ported over _d_arrayctor, _d_arraysetctor, _d_arraycatT, and
  _d_arraycatnTX to templates.
- Implemented lowering code for _d_arrayctor, _d_arraysetctor.
  - The whole functions for these could be moved to a template function,
    so these no longer depend on TypeInfo.
- Worked on the lowering code for _d_arraycatT and _d_arraycatnTX.
  - Debating if _d_arraycatT should be dropped and only use
    _d_arraycatnTX, to help with the lowering code.


Next week
~~~~~~~~~

- Finish the _d_arraycat{T,nTX} hook.
  - Probably merge _d_arraycatT into _d_arraycatnTX to help the lowering
    code.
- Make and submit the PRs for _d_array*ctor and _d_arraycat{T,nTX}.
- Move the assignment of _d_arrayappendcT back outside the hook to fix
  the problem.


Blockers
~~~~~~~~

As mentioned in /This week/ about _d_arrayappendcT, the problem is that
phobos unittests passes, even if the unittest works for druntime.  The
only thing I can think of is that the postblit is called before the
object is sent to _d_arrayappendcT instead of after the
assignment. Moving the assignment back outside of _d_arrayappendcTX
should fix this, to restore what is currently done with the hook.

I have not focused too much time on this issue because I wanted to get
the _d_array*ctor and _d_arraycat* up and running first as I did not
know what problem I would encounter with these.

--

The lowering code for _d_arraycat{T,nTX} is currently pretty ugly
because I need to handle many cases of the lowering. For example:

 ```d
 S[2] a;
 S[] a4 = a ~ a ~ a ~ a;
 // Step 1:
 S[] a4 = _d_arraycatT(a, a) ~ a ~ a;
 // Step 2:
 S[] a4 = (S[][3] __cat2 = [a, a, a];) , _d_arraycatnTX(cast(S[][])__cat2)  ~ a;
 // Step 3:
 S[] a4 = (S[][4] __cat4 = [a, a, a, a];) , _d_arraycatnTX(cast(S[][])__cat4);
 ```
(This will '__cat*' is used to not allocate memory heap and break the
'scope' requirement code.)

The rewrites steps are like this because it works from the inside
outwards compared to the outside inwards, like what is done in e2ir.d.

As mention in /Next week/ I will merge _d_arraycatT into nTX to help
with all the different edge cases and to get more manageable and
reviewable code.


Week 4
======

This week
~~~~~~~~~

- Worked on more _d_arrayappend{T,cTX}, _d_arraycatnTX and
  _d_array{set,}ctor.
- I got _d_arrayappend{T,cTX} working after moving the assignment back
  outside of the _d_arrayappendcTX call.

dmd PR: [https://github.com/dlang/dmd/pull/9982]
druntime PR: [https://github.com/dlang/druntime/pull/2632]

- I've submitted PR for _d_arraycatnTX, even if the dmd lowering code is
  not working right now.
dmd PR: [https://github.com/dlang/dmd/pull/10064]
druntime PR: [https://github.com/dlang/druntime/pull/2648]


Next week
~~~~~~~~~

Rework my timetable to see if I will be able to finish the rest of the
hook for the allocated timeslots. The rework is needed because
_d_arraycat{T,nTX} and _d_array{set,}ctor are still not done due to bugs
that are hard to understand and fix. The goals is to with my acquired
knowledge of what I've done now be able to better classify the
difficulty level of each hook. As I've really realized that the
line-of-code metric was a really bad one that did not at-all show the
difficulty level of the hooks.

Continue the work on _d_arraycat{T,nTX} and _d_array{set,}ctor to make
sure that they are finish before the second evaluation. I will probably
have to put these temporary on the backlog while figuring out how hard
the next hook will be to port, which will probably be
_d_arraysetlength{,i}T.


Blockers
~~~~~~~~

As mentioned before I had postblit problems with the previous
_d_arrayappendcT function, but now when I moved the assignment outside
it passes all the unittests.

_d_arraycatnTX is currently not working because my lowering code somehow
send in a temporary incorrectly into a hook, and it also break a assert
inside the dmd backend code. More information about this can be found
inside the dmd PR.

The last hooks that isn't done is the _d_array{set,}ctor, here I still
have left to finish the logic for when to write AssignExp expressions,
as some of the expression should not be rewritten. I also need to add
logic to the hook for when to export the interface as nothrow. If I
force it to be nothrow this unittest will fail:
[https://github.com/dlang/druntime/blob/9f44ee59ac784e28c19631dd0be2b868766f4348/src/object.d#L3538]
But if I don't force it, it will always default to throw and it will
break code that calls this hook from nothrow scopes. So I need to
implement some logic that determine when to export as nothrow and when
to export it as throw.


Reflections on progress so far
==============================

Now that the first evaluation is in progress I wanted to just write
about what my thoughts are after working on three hooks during these
four weeks. I added this section to add some transparency of what is
happening behind the scenes of my GSoC work, and to document everything
in case it could be good to know. I also enjoy being critical about what
I have done to get a teachable moment out of it.


Planning
~~~~~~~~

The metric I used in my proposal, lines of code, was a really bad
choice. I've realized that it is not the amount of code that the hook is
called that describes its difficulty but than in what context it is
called and with that data it is called with. I have started to going
through the rest of the hooks I have not started with to try and build
up new metrics that I can use to better evaluate the difficulty of the
hook and to make sure I allocate enough time to be able to success with
the translation within the allocated timeslots.

Because I did not fully realize the difficulty, i.e. possible blockers,
of the hooks so far I have not been able to finish all of them within
the timeslot I wanted. I got the array append hook working, but it was
in the last minute before the week four ended.

With the knowledge of everything I did not previously know, did not plan
for, or even maybe overlooks, I want to be able to in the future of GSoC
identify these problem and know what to do to fix them.

On Wednesday, 26 June 2019 at 01:49:37 UTC, Dan Printzell wrote:
> Reflections on progress so far
> ==============================
>
> Now that the first evaluation is in progress I wanted to just write
> about what my thoughts are after working on three hooks during these
> four weeks.

Thank you for sharing this. I find this project very interesting.
IMO, all you wrote is valuable and gives some tips for the future.

Cheers,
Piotrek

On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
> [...]

I forgot to post week 5 & 6, so here is week 5-7

Week 5
======

This week
~~~~~~~~~
- Translate _d_arraysetlengthT to template
- _d_arraysetlengthT druntime patch got merged
  - [https://github.com/dlang/druntime/pull/2656]
- Got _d_arraycatnTX to pass all the testcases.
- Updated all the PRs based on feedback.

Current PRs: _d_arraysetlengthT:
- [https://github.com/dlang/dmd/pull/10106]

_d_array{,set}ctor:
- [https://github.com/dlang/druntime/pull/2655]
- [https://github.com/dlang/dmd/pull/10102]

_d_arraycatnTX:
- [https://github.com/dlang/druntime/pull/2648]
- [https://github.com/dlang/dmd/pull/10064]

_d_arrayappend{T,cTx}:
- [https://github.com/dlang/druntime/pull/2632]
- [https://github.com/dlang/dmd/pull/9982]


Next week
~~~~~~~~~
- Work on the feedback I get on the dmd patches, when the druntime patch
  have been merged.
- Start working on the _d_arrayliteralTX hook, which was planned for
  next week.


Blockers
~~~~~~~~
Catching `Throwable` and then rethrowing makes the function not be able
to be deduced as `nothrow`.

Realized that there is no way to easily check if something is `nothrow`
using only druntime. Except for `__traits(getFunctionAttributes, x)`,
but that requires a lot of code just to check for nothrow.


Week 6
======

This week
~~~~~~~~~
Made two fix PR for `_d_arraysetlengthT*` to disallow the hooks to be
inlined, and to force it to generate a *Trace instant for each regular
hook. Both of these PRs have been merged.

Updated `_d_array{,set}ctor` based on feedback from thewilsonator.
[https://github.com/dlang/druntime/pull/2655#issuecomment-507923713]

`_d_arraycatnTX` is updated with the sane fixes as `_d_arraysetlengthT`.
[https://github.com/dlang/druntime/pull/2648]

Next week
~~~~~~~~~
- Porting over on `_d_arrayliteralTX`.
- Apply fixes to `_d_arrayappendcTX`.
- Work on getting the previous PRs merged.

Blockers
~~~~~~~~
Learned that I need `pragma(inline, false);` for the hooks to not break
my CTFE interception, and the bigger thing I had to fight with was
getting the compiler to instantiate both `_d_arraysetlengthT` and
`_d_arraysetlengthTTrace`. Here I tested multiple thing, like
referencing the trace function from the regular hook, but this did not
work and only caused template instantiating problems. What I used in the
end, which was also the most simple and clean solution, was to wrap both
hooks inside of a `template _d_arraysetlengthTImpl` block. This caused
both functions to be instantiated, even if only one hook was referenced.

All of this was needed because now if you, for example, use
`std.array.Appender!(char[])` which is a template struct, in a project
when you compile with the `-profile=gc`. Means that the template gets
instantiated but because that template instantiating already exist
inside of phobos, dmd chooses to use that one instance. But because that
phobos instance didn't use `_d_arraysetlengthTTrace`, it cause a
`unknown reference to`.  I'm not sure if I explain the problem here
correctly, but it was weird and hard to understand enough to be able to
make a fix for it.

_d_arraysetlength's dmd PR is currently failing the CI during the
`profilegc`, but I cannot reproduce it locally.


Week 7
======

This week
~~~~~~~~~
Continued to work on my PR to make them into a mergeable state.

'Cleanup and fix implementation of _d_array{setlengthT,catnTX}Trace'
- [https://github.com/dlang/druntime/pull/2673]
Fixes some problem with the Trace entrypoint for hooks not calling
`rt.profilegc.accumulate`, and a refactoring to templateize the Trace
function to make sure that code isn't duplicated over all the Trace
entrypoints.

Finished the dmd PR for _d_arraysetlengthT using all the fixes I've been
working on the last week.  It currently waits for someone to do the last
review and hopefully merge it.
[https://github.com/dlang/dmd/pull/10106]

Next week
~~~~~~~~~
Pretty much same as last week:
- Finish port of `_d_arrayliteralTX`
- Work on getting the previous PRs merged.
  - The pr for _d_arrayappend* needs to be rebased and integrated with
    the new way to implement Trace
    function. [https://github.com/dlang/druntime/pull/2632] Its dmd
    patches also need to be turned into a PR.
  - [https://github.com/dlang/dmd/pull/10102] - _d_array{,set}ctor and
    [https://github.com/dlang/dmd/pull/10064] - _d_arraycatnTX needs to
    be updated to follow how _d_arraysetlengthT was implemented.

Blockers
~~~~~~~~
The previous week I've learned more about every edge-case that I've
encountered, and how to fix them.  Like the how I fixed and cleaned up
the *Trace implementations, here I had to figured out how to do a `pure`
bypass, while still not allowing the compiler to remove calls that are
requested, but the optimizer thinks are useless. I solved this by `if
(...) assert(0);`, with a condition that I know will never pass, and the
optimizer will never remove a `assert(0)`, which forced it to allow the
call to still be done.

During the creation I found a way inside of the dmd's source-code on how
to bypass pure, without using any `cast()`s. Sadly this was classified
by github user aG0aep6G as a bug. Personally I'm a bit sad that this
bypass method was removed in [https://github.com/dlang/dmd/pull/10172],
as I really though it looked cleaner than using casts, but as this is a
bug I cannot argue against removing it.

On the note about bypassing pure I've mentioned that I will probably
introduce `HookCallExp` or a bool inside of `CallExp` to allow purity to
be ignore, but only for hook calls. If/when I do this I also want to add
support to be able to detect hooks that have been inlined, maybe by
adding a `HookCompoundStatement` and/or `HookCommaExp`. But currently I
want to spend my time to get all the hooks into dmd and druntime, and
probably after GSoC start introducing these improvements.

Another problem I encountered after the `HookTraceImpl` PR was merged
was that I broke the CTFE intercept code I had for *Trace hooks. This
happened because the following code does not introduce a new symbol into
the context and it is just alias:
,----
| alias _d_arraysetlengthTTrace = HookTraceImpl!(Tarr, _d_arraysetlengthT, errorMessage);
`----
This is something I overlooked when doing the PR, but in the end this
was not hard to big and I would say actually lead to better code. As
now I have a `removeHookTraceImpl` function inside dinterpret.d:
[https://github.com/dlang/dmd/pull/10106/files#diff-264b29261e215213d3026cd44003c2ebR7452]
,----
| //This function will replace:
| HookTraceImpl!(T, Hook, errMsg).HookTraceImpl(..., parameters)
| // with
| Hook(parameters)
`----
It does this by getting getting the `FuncDeclaration` which in this case
is `HookTraceImpl` function, goes to its parent the `HookTraceImpl`
template, takes the second template parameters, and uses to rewrite the
`CallExp` and replace the `FuncDeclaration` that is used to do the CTFE
interception.

On Wednesday, 17 July 2019 at 17:04:45 UTC, Dan Printzell wrote:
> It does this by getting getting the `FuncDeclaration` which in this case
> is `HookTraceImpl` function, goes to its parent the `HookTraceImpl`
> template, takes the second template parameters, and uses to rewrite the
> `CallExp` and replace the `FuncDeclaration` that is used to do the CTFE
> interception.

Out of curiosity, what is CTFE interception?

On Wednesday, 17 July 2019 at 20:47:02 UTC, Olivier FAURE wrote:
> On Wednesday, 17 July 2019 at 17:04:45 UTC, Dan Printzell wrote:
>> It does this by getting getting the `FuncDeclaration` which in this case
>> is `HookTraceImpl` function, goes to its parent the `HookTraceImpl`
>> template, takes the second template parameters, and uses to rewrite the
>> `CallExp` and replace the `FuncDeclaration` that is used to do the CTFE
>> interception.
>
> Out of curiosity, what is CTFE interception?

Basically I rewrite the expression `arr.length = len;` into a function
call to the templated runtime function `_d_arraysetlengthT(arr,
len);`. But because of this the CTFE can no longer recognize that the
code is trying to change the size of `arr`, only that it is calling a
function called `_d_arraysetlengthT`.
The CTFE intercept is basically intercepting all the function call
that the CTFE would run and checks if it is trying to call the
`_d_arraysetlengthT`, and if it is it will rewrite the expression back
to `arr.length = len` and then interpret that expression instead of
the function call.

This is needed as the CTFE way of resizing arrays is not the same as
how regular code resizes arrays.

On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
> [...]

Week 30
=======

This week
~~~~~~~~~
The translation of _d_arraysetlengthT is fully merged!

Worked on getting the _d_arrayappend{T,cTX} and _d_array{,set}ctor dmd
PRs into a mergeable state, currently stuck at getting it to pass the
testcases. More information in Blockers.

I never finished the _d_arrayliteralTX PRs as I wanted to get the other
PR working first, but I never thought they would take this long time to
fix.

Next week
~~~~~~~~~
Pretty much same as last week:
- Finish port of `_d_arrayliteralTX`
- Start working on the `_d_delarray_t*`, `_d_arrayassign*`, and
  `_d_arraysetassign` translations.
- Work on getting the previous PRs merged.
  - The PR for _d_arrayappendcTX need to more logic on how to lower the
    appending, and more logic on how to interpret the lowered
    expressions.
  - _d_array{,set}ctor need to implements some fixes to get the CTFE to
    work correctly.
  - _d_arraycatnTX need fixes related to the CTFE interpretion. Also
    need to fix the code based on the received feedback.


Blockers
~~~~~~~~
This passed week I've been fighting with the optimizer to always be able
to detect the lowered code inside dinterpret.

With _d_arrayappendcTX I've (so far?) identified four different
expression groups that I need be able to detect and rewrite back so the
CTFE interpretion works correctly. They are:

```
//Case 1: left is the _d_arrayappendcTX call, right is the assignment onto the array.
trs ~= fork(t, next, t.counter); // =>
_d_arrayappendcTX(trs, 1LU) , trs[__dollar - 1LU] = fork(t, next, t.counter);

//Case 2: left is the _d_arrayappendcTX call, right is the construction onto the array.
arr ~= T(nanF, '\xff').this(0); // =>
_d_arrayappendcTX(arr, 1LU) , (arr[__dollar - 1LU] = T , arr[__dollar - 1LU].this(0));

//Case 3: left is the _d_arrayappendcTX call, right is a postblit and the assignment onto the array.
this.data ~= ((T __copytmp = (__copytmp = val).__fieldPostblit();) , __copytmp); // =>
_d_arrayappendcTX(this.data, 1LU) , ((T __copytmp = (__copytmp = val).__fieldPostblit();) , this.data[__dollar - 1LU] = __copytmp);

//Case 4: left is __appendtmp var, right is case 1
op ~= cast(immutable(char))this.x; // =>
(immutable ref immutable(char) __appendtmp115 = this.x;) , (_d_arrayappendcTX(op, 1LU) , op[__dollar - 1LU] = __appendtmp115);
```

Related to the _d_array{,set}ctor PR, it currently fails at object.d
line 3636, 'assert(postblitRecurseOrder =' order);=. This is because
order is equal to:
```
copy inner #1
copy inner #1
copy inner #1
copy outer
copy inner #1
copy inner #2
destroy inner #1
```
Verses what is expected:
```
copy inner #1
copy inner #1
copy inner #1
copy outer
copy inner #1
copy inner #2
destroy inner #1
destroy outer
destroy inner #1
destroy inner #1
destroy inner #1
```

It has something to do with optimizations, and probably exceptions. I
have not been able to find the cause of it yet. But what happens is that
it never calls the destructor for 'arrCopy[0]'.

Related to _d_arraycatnTX it currently broken two place so far which I
need to find a real fix to before it is done. The first one is that
implicit cast to 'const(char)*' does not work anymore of a string
concatenations. The second problem I've found is that:
`void badOp(int x, int y = 1 ~ "string") {}` prints the error:
`Error: cannot implicitly convert expression ['\x01', 's', 't', 'r', 'i', 'n', 'g'] of type string to int`
Instead of: `Error: cannot implicitly convert expression "\x01string" of type string to int`

I'm not sure why it prints the string as a 'char[]', but it needs to be
fixed before it can be merged.


Thoughts after the second milestone
===================================
From my point of view I would classify this milestone as less productive
compared to the first one, but from the sense that I got a hook fully
into dmd and druntime you could classify that this was more successful
milestone.

I need to figure out what is most important to get done for the final
milestone and prioritize that. I don't think I will be able to finish
everything 100% as the dmd PR are getting more difficult for
optimization, and each edge-case. But I do think I will be able to
finish all the druntime PRs.

On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
> [...]

Sorry, I've been busy with getting everything into a workable to state
so I've been late to with posting these updates. So this time it is for
the last three weeks. There is also some planning for the final
submission at the bottom.


Week 31
=======

This week
~~~~~~~~~
Last week I've been trying a lot to use the `__ctfe` variable to detect
when to call the rewritten hook, verses when to call the original
expression. As discussed, the old CTFE interception turned into a big
edge-case friendly place where it needs to be able to detect all the
different ways the AST could look when executing a hook.

Sadly lowering `this.data ~= val` to ` __ctfe ? this.data ~= val :
_d_arrayappendT(this.data, val)` didn't work as it threw the following
assertion in the backend. `dmd: dmd/backend/symbol.d:1170: Assertion
'(*s).Ssymnum == -1' failed.`.

I have not really made any progress on the next hooks or the previous
hooks as I wanted to get the DMD part done first. It is in my opinion
pretty useless to add translated hooks into druntime if their dmd part
won't work. As adding unused code is still adding maintenance cost to
druntime.


I've marked all the current dmd PR as WIP, as they should all move to a
better way of detecting CTFE. I've also update
[https://definewild.se/public/Hooks.html] to reflect the current state
of all the hooks.

Next week
~~~~~~~~~
- Try and figure how to use CondExp, if it is even possible.
- Maybe rethink my time plan. Should I focus on getting more druntime
  hook merged, or should I focus on getting the current merged hooks
  working? (The later is probably the better choice.)

Same as last week:
- Finish port of `_d_arrayliteralTX`
- Start working on the `_d_delarray_t*`, `_d_arrayassign*`, and
  `_d_arraysetassign` translations.
- Work on getting the previous PRs merged.
  - The PR for _d_arrayappendcTX need to more logic on how to lower the
    appending, and more logic on how to interpret the lowered
    expressions.
  - _d_array{,set}ctor need to implements some fixes to get the CTFE to
    work correctly.
  - _d_arraycatnTX need fixes related to the CTFE interpretion. Also
    need to fix the code based on the received feedback.

Blockers
~~~~~~~~
The assert problem seem to have something to do with the `val` part
getting rewritten. Maybe dmd doesn't support DeclarationExp inside of
CondExp?

```
e2ir.d, visit(CondExp de): __ctfe ? this.data ~= ((InversionList!(GcPolicy) __copytmp4144 = (__copytmp4144 = val).__fieldPostblit();) , __copytmp4144) : ((InversionList!(GcPolicy) __appendtmp4145 = (InversionList!(GcPolicy) __copytmp4144 = (__copytmp4144 = val).__fieldPostblit();) , __appendtmp4145 = __copytmp4144;) , (_d_arrayappendcTX(this.data, 1LU) , this.data[__dollar - 1LU] = __appendtmp4145) , this.data)
e2ir.d, visit(CommaExp ce): (InversionList!(GcPolicy) __copytmp4106 = (__copytmp4106 = val).__fieldPostblit();) , this.data[__dollar - 1LU] = __copytmp4106
e2ir.d, visit(DeclarationExp de): (InversionList!(GcPolicy) __copytmp4106 = (__copytmp4106 = val).__fieldPostblit();)
dmd: dmd/backend/symbol.d:1170: Assertion `(*s).Ssymnum == -1' failed.
```



Week 32
=======

This week
~~~~~~~~~
*_d_arrayappend{T,cTX}:* Required druntime fixes:
[https://github.com/dlang/druntime/pull/2718] Update dmd patches:
[https://github.com/dlang/dmd/pull/9982] This uses the new `__ctfe ?
orgExp : loweredExp` technique instead of intercepting CTFE calls. I
also added the fix to the assert
[https://github.com/dlang/dmd/pull/9982/commits/2d81b673424175ae5304f2eee7dcb7bd87d43cc3]
And I made the backend only compile the `!__ctfe` path to machine code,
to be able to `assert(0)` the calls to the old hook functions:
[https://github.com/dlang/dmd/pull/9982/commits/1f4d9bf17033cee8e2458c3d1c35a52632caa6f8]

It took some fighting to get the hook working, because I triggered some
asserts in the codegen. But I was able to fix / work around all of them.

Next week
~~~~~~~~~
- Update `_d_arrayappend{T,cTX}` PRs based on feedback.
- Fix `_d_array{,set}ctor`.
- Fix `_d_arraycatnTX`.

- Figure out what is needed for the final submission.

Blockers
~~~~~~~~
The current problem with `_d_array{,set}ctor` is that it doesn't call
all the dtors. It expects to find this in the output `order`:

```
    copy inner #1
    copy inner #1
    copy inner #1
    copy outer
    copy inner #1
    copy inner #2
    destroy inner #1
    destroy outer
    destroy inner #1
    destroy inner #1
    destroy inner #1
```
But only this is found:
```
    copy inner #1
    copy inner #1
    copy inner #1
    copy outer
    copy inner #1
    copy inner #2
    destroy inner #1
```

Unittest:
[https://github.com/dlang/druntime/blob/master/src/object.d#L3647]

I think the problem is that DeclarationExp/VarDeclaration, doesn't
handle `scope (failure)` correctly. Currently I'm only rewriting the
AssignExp that is inside the `VarDeclaration`. My current theory is that
if I rewrite the DeclarationExp instead the hook may start working
against.


I have not look into `_d_arraycatnTX` issues this week. But using the
new `__ctfe ? :` way of restoring the CTFE functionality should
hopefully allow me to remove some hacks I had to implement. Which in
turn will allow the PR to not break any existing code.



Week 33
=======

This week
~~~~~~~~~
*_d_arrayappend{T,cTX}:* Remove the debug `message()` calls.

*_d_array{,set}ctor:* Fixed the hook by doing a weird hack I don't fully
understand why it is needed. Not sure if it is a dmd bug that could
happen in usercode or if it is just because it is called inside of a
`DeclarationExp._init`.
[https://github.com/Vild/druntime/blob/FixArrayCtorHook/src/core/internal/array/construction.d#L59]

I have not PR this yet as I'm currently depending on the PR I submitted
last week related to _d_arrayappend,
[https://github.com/dlang/druntime/pull/2718]. I may merge these changes
into that PR.

*_d_arraycatnTX:* I've figured out the currently problem with why cannot
compile this:
[https://github.com/dlang/dmd/blob/b2522da8566783491648bc104a29b42dc2dc569e/src/dmd/mars.d#L2174]

It is because the somewhere the `commited` member of StringExp get set
to `1` even if the CTFE was able to run the call at compile and get a
single StringExp out of it.

Next week
~~~~~~~~~
- Fix the last changed to `_d_arraycatnTX`.
- Get `_d_arrayappend{T,cTX}`, `_d_array{,set}ctor`, and
  `_d_arraycatnTX`.
- Work on the writeup on what I've learned etc, that I lifted last week
  update.

Blockers
~~~~~~~~
*_d_array{,set}ctor:* Here is just some of the differences I found in
GHIDRA when I compared compile with and without the hack.  The
differences I could find of the hook is that the working one have two
try-blocks ([https://i.definewild.se/3377]) and the loop looks a bit
different ([https://i.definewild.se/165e] left is non-working, right is
working).

*_d_arraycatnTX:* The biggest blocker right now is how I should solve
the `_d_arraycatnTX` problem. But I don't think it will be too bad, as
now I've been able to identify where the problem is. Now I just need to
compare what code paths the different concatenations goes though, and
why they differ.

```
enum fourfivesix = "456";
enum world = () {
    string s;
    s ~= "world!";
    return s;
}();
void test(const(char)* fmt) {}
void main() {
    test("derp" ~ " herp"); // Works
    test("123 " ~ fourfivesix); // Works
    test("hello " ~ world); // Doesn't
    /* Error: function cattesting.test(const(char)* fmt) is not callable using argument types (string)
     *      cannot pass argument "hello world!" of type string to parameter const(char)* fmt
     */
}
```



Planning for final submission
=============================

I will not be able to finish the hooks all the hooks I listed in my
proposal. Instead I should probably just focus on finishing the current
WIP hook: `_d_arrayappend{T,cTX}`, `_d_array{,set}ctor`, and
`_d_arraycatnTX`. If I finish these I will have successed with
translating four out of nine hooks (the forth hook is _d_arraysetlength
which is already merged).

Furthermore as a final submission it probably should include some
documentation on how to do the translation process, to help aid others
to translate hooks. This could include the following sections/topics:
- Finding a hook to translate
- Where to add the new code, how to update the buildsystem
- How to manage the CTFE.
- Navigating the codebase: Which dmd files are relevant? Where should I
  look for class X?
- Examples of previous translations
  - with notes on how they were updated / fixed
- Debugging technique
- How to implement a -profile=gc entrypoint.

I'm not sure where the best place for this would be as this is more for
new compile developers than the average D user. I guess the best place
would be the wiki.