April 29, 2012
http://d.puremagic.com/issues/show_bug.cgi?id=7442



--- Comment #10 from Don <clugdbug@yahoo.com.au> 2012-04-29 00:02:21 PDT ---
(In reply to comment #7)
> I investigated this further and conclude that there are 2 factors at work.
> 
> I removed few thousands of codepoints from Letter, so it doesn't run out of RAM
> outright.(see code below)
> Also separated parse from build steps.
> 
> Here are collected stats on CTFE.
> 
> parse only:
> 
>         ---- CTFE Performance ----
> max call depth = 20     max stack = 44
> array allocs = 2761     assignments = 430837
> 
> build:
>         ---- CTFE Performance ----
> max call depth = 20     max stack = 73
> array allocs = 8264     assignments = 1293421
> 
> Parsing creates all the datastructures for unicode table machinery
> it takes slightly less then half of all allocs already.
> Another thing to notice is it fetures higher allocations per assigment.
> 
> Then comes the codegen step and it's CTFE only and far more alloc happy.
> Frankly I see no way to reduce all of this alloc fun because of COW
> that will ruin any attempt to preallocate buffer for generated code.
> Am I right that arrays do dup on every write?

No, that was true a year or so ago, but not any more. One case which still causing array dups is comparing slices of arrays (eg, if (x[0..5] == y). I've put the machinery in place to get rid of that now (it's not gone yet, I need another simple pull request for that). That _might_ help a little bit.

Slicing also causes array dups with ~, eg z = x[0..5] ~ y; creates a temporary array with x[0..5]. But those are the only two that I know of.

Clearly in this case, it's slow because it does 1.3 million assignments. It's a duplicate of bug 6498, unless you can find some way of drastically reducing the work it has to do.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
April 29, 2012
http://d.puremagic.com/issues/show_bug.cgi?id=7442



--- Comment #11 from Dmitry Olshansky <dmitry.olsh@gmail.com> 2012-04-29 05:53:34 PDT ---
> 
> No, that was true a year or so ago, but not any more. One case which still causing array dups is comparing slices of arrays (eg, if (x[0..5] == y). I've put the machinery in place to get rid of that now (it's not gone yet, I need another simple pull request for that). That _might_ help a little bit.
> 
> Slicing also causes array dups with ~, eg z = x[0..5] ~ y; creates a temporary array with x[0..5]. But those are the only two that I know of.
> 

Ok does a ~= b; *always* reallocate a ? If not I'm fine.

> Clearly in this case, it's slow because it does 1.3 million assignments. It's a duplicate of bug 6498, unless you can find some way of drastically reducing the work it has to do.

Well take a look at "benchmark" for CTFE. It does a lot of bit-setting on array of bytes. Posiibly a source of asigments since it does 1 bit-set per codepoint. That's unavoidable I'm afraid.

I may special case things somewhat so that it checks if fullword write of zeros or ones is possible.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
March 14, 2013
http://d.puremagic.com/issues/show_bug.cgi?id=7442



--- Comment #12 from Dmitry Olshansky <dmitry.olsh@gmail.com> 2013-03-14 11:53:00 PDT ---
(In reply to comment #0)
> Test case:
> 
> ------------------------
> import std.regex;
> enum bug7442 = ctRegex!`\p{Letter}`;
> ------------------------
> 
> Compile with `dmd -c test7442.d`. DMD will very quickly use up all system memory. (I'll try to reduce the test case later.)

Now this compiles for me in 2.063:

import std.regex;
enum bug7442 = ctRegex!`\p{Letter}`;

------------------------
Total time (ms): 11197.2

And even both cases combined:

import std.regex;
enum bug7442 = ctRegex!`\p{Letter}`;
void wcpx(string fn)
{
    enum ctr =  regex(r"\w+","g");
}

------------------------
Total time (ms): 17443.3

The latter takes around 600Mb virtual memory as measured on Win7 x64 with 32-bit executable. DMD lately has been tweaked to actually use up to 2GB of memory on win32.

And the time is not all that bad.

Should we close this as the root case of improving dmd's CTFE memory footprint is covered elsewhere? Just asking.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
June 07, 2013
http://d.puremagic.com/issues/show_bug.cgi?id=7442


Jameson <beatgammit@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |beatgammit@gmail.com


--- Comment #13 from Jameson <beatgammit@gmail.com> 2013-06-06 23:21:22 PDT ---
(In reply to comment #12)
> (In reply to comment #0)
> > Test case:
> > 
> > ------------------------
> > import std.regex;
> > enum bug7442 = ctRegex!`\p{Letter}`;
> > ------------------------
> > 
> > Compile with `dmd -c test7442.d`. DMD will very quickly use up all system memory. (I'll try to reduce the test case later.)
> 
> Now this compiles for me in 2.063:
> 
> import std.regex;
> enum bug7442 = ctRegex!`\p{Letter}`;
> 
> ------------------------
> Total time (ms): 11197.2
> 
> And even both cases combined:
> 
> import std.regex;
> enum bug7442 = ctRegex!`\p{Letter}`;
> void wcpx(string fn)
> {
>     enum ctr =  regex(r"\w+","g");
> }
> 
> ------------------------
> Total time (ms): 17443.3
> 
> The latter takes around 600Mb virtual memory as measured on Win7 x64 with 32-bit executable. DMD lately has been tweaked to actually use up to 2GB of memory on win32.
> 
> And the time is not all that bad.
> 
> Should we close this as the root case of improving dmd's CTFE memory footprint is covered elsewhere? Just asking.

This now also compiles for me. I haven't tested the memory usage very well, but it seems to only use about 400mb or so on my machine, which is completely reasonable.  Time is consistently around 3.5 seconds, so I'm happy.

Platform: Arch Linux x86_64.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
1 2
Next ›   Last »