Update for week 3 of Milestone 3

- Fixed an issue which deals with the way nested aggregates are used outside of their parent (enclosing) aggregate

e.g.:

struct A {
    struct B {
        int a;
        int c;
    } b_obj;
    int other;
};

void f(struct B*); // works in C, in D it should be A.B (but it was still translated to just B)
// Also applicable to field types

A better explanation and an actual solution are presented in the PR below.
PR: https://github.com/atilaneves/dpp/pull/226

- Did some profiling on DPP with lots of kernel header files included in the same .dpp file.
Memory usage after libclang finishes parsing: 300MB.
Memory usage after DPP finishes processing the AST provided by libclang: 6-7GB (this is where some fishy things happen).
DPP then tries to use 'cpp' on the .d.tmp file for C preprocessing, but the process spawning fails, as there is not enough memory left to fork and execute (I was testing on an 8GB machine).

It turns out (and huge thanks to Edi and Razvan for your help with this) that the Appender object  '_lines' (which stores the strings to be written in the translated D file) leaks memory. This was apparently an already known issue: https://issues.dlang.org/show_bug.cgi?id=19511

I've changed the type from Appender to built-in array and the maximum memory usage at any point is somewhere between 2.0-2.4GB, way better than 7GB :)

Edi also wrote a simple test and it seems that for appending ~ 10^6 strings of length 20 each, Appender is just 20ms faster. I also tested this with headers generating a 250K line (850K words total) D file and performance doesn't differ (I also think that as long as Appender is leaking it shouldn't be used no matter the speed boost, considering the large C codebases that DPP could be used with).
PR: https://github.com/atilaneves/dpp/pull/227

Update for week 4 of Milestone 3

- For the memory usage issue: I previously proposed changing from Appender to array (as I've seen the memory dropping from ~6GB to ~2GB). In the last week I've been continuously testing those two versions to be sure. However I don't get constant results: sometimes both the appender and the array use more than 5GB, and other times both use 2GB (once I even got to max 600MB). The tests were done with the same input file and after system restart (just to make sure it's not a cache thing of some sorts). If you have any guess about why this might happen, please feel free to reply and let us know.

I also tested the performance for Appender vs array on Python.h (which is way smaller than the kernel headers I tested above), as Atila requested in the PR.

Appender version (~230MB):

User time (seconds): 18.78
System time (seconds): 2.62
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.41

Array version (~240MB):

User time (seconds): 17.39
System time (seconds): 0.22
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:17.61

- I've identified the issue which inserted C warnings in the generated D code (it's the C preprocessor whose output is redirected into the file). I have a local fix for this (redirecting the error messages), but I didn't make a PR yet.

- Also locally fixed an issue with void type extern variables. As discussed with my mentors, the solution is to change from void to char*.

Update for week 1 of Milestone 4

- Solved and made a PR for the clang warnings being written to the .d file: https://github.com/atilaneves/dpp/pull/232

- Solved and made a PR for the issue of variables of type void in C: https://github.com/atilaneves/dpp/pull/233

I had some issues running the unittests locally and I've been trying to debug that while working on the above PRs. I decided to push the changes anyway, but I think that in the meantime Atila managed to fix those issues.

In the next 2 weeks I won't be able to consistently work on the project as before, so I will probably be posting the updates for weeks 2 and 3 of Milestone 4 as part of the update for week 3.