I'm working on my parser generator and trying to shoe horn it into a smaller memory foot print. One approach I'm trying, is to replace recursive templates with CTFE function and string mixins. This is in an effort to reduce the number of types that get declared (mostly I'm using struct templates). The problem is that the CTFE functions seem to actually use MORE ram than the templates do. I've tried a number of things like converting to an O(n log n) concatenation system rather than a O(n^2) and other such tricks but everything seems to be going backwards. Any advice? (D v1 BTW) I can post code if anyone cares (it's kind of large: 1.2KLOC, takes a while to compile: ~6min, and use a lot of RAM: ~2GB).

(after a bit more fiddling)

I've got it down to 1GB of ram (Yeah!). My conclusions: don't build strings with CTFE unless it's a lot harder to do it with templates. It seems that CTFE is a bigger memory hog than templates even with temples memoizing everything into the symbol tables.

BCS wrote:
> I'm working on my parser generator and trying to shoe horn it into a smaller memory foot print. One approach I'm trying, is to replace recursive templates with CTFE function and string mixins. This is in an effort to reduce the number of types that get declared (mostly I'm using struct templates). The problem is that the CTFE functions seem to actually use MORE ram than the templates do. I've tried a number of things like converting to an O(n log n) concatenation system rather than a O(n^2) and other such tricks but everything seems to be going backwards. Any advice? (D v1 BTW) I can post code if anyone cares (it's kind of large: 1.2KLOC, takes a while to compile: ~6min, and use a lot of RAM: ~2GB).
> 
> (after a bit more fiddling)
> 
> I've got it down to 1GB of ram (Yeah!). My conclusions: don't build strings with CTFE unless it's a lot harder to do it with templates. It seems that CTFE is a bigger memory hog than templates even with temples memoizing everything into the symbol tables.
> 
> 

Yes I believe Don has previously reported that CTFE manipulations on strings basically *never* free the intermediate strings.  Basically the CTFE engine needs a garbage collector (or equiv).

--bb

On Fri, 14 Mar 2008 08:42:27 +0900, Bill Baxter wrote:

> BCS wrote:
>> I'm working on my parser generator and trying to shoe horn it into a smaller memory foot print. One approach I'm trying, is to replace recursive templates with CTFE function and string mixins. This is in an effort to reduce the number of types that get declared (mostly I'm using struct templates). The problem is that the CTFE functions seem to actually use MORE ram than the templates do. I've tried a number of things like converting to an O(n log n) concatenation system rather than a O(n^2) and other such tricks but everything seems to be going backwards. Any advice? (D v1 BTW) I can post code if anyone cares (it's kind of large: 1.2KLOC, takes a while to compile: ~6min, and use a lot of RAM: ~2GB).
>> 
>> (after a bit more fiddling)
>> 
>> I've got it down to 1GB of ram (Yeah!). My conclusions: don't build strings with CTFE unless it's a lot harder to do it with templates. It seems that CTFE is a bigger memory hog than templates even with temples memoizing everything into the symbol tables.
>> 
>> 
>> 
> Yes I believe Don has previously reported that CTFE manipulations on strings basically *never* free the intermediate strings.  Basically the CTFE engine needs a garbage collector (or equiv).
> 
> --bb

Looks like D needs a script engine for CTFE
that makes use of a GC. Would be a good feature imho.

Reminds me of Virgil when I heard about in a post some time ago. Virgil uses a compile time GC that includes reachable memory into the binary for some kind of memory pre-allocation. http://compilers.cs.ucla.edu/virgil/ref/pubs.html

my2cents

"Bill Baxter" <dnewsgroup@billbaxter.com> wrote in message news:frce57$1vf5$1@digitalmars.com...
>
> Yes I believe Don has previously reported that CTFE manipulations on strings basically *never* free the intermediate strings.  Basically the CTFE engine needs a garbage collector (or equiv).
>
> --bb

You know, if we had a D compiler made in D, it'd be GCed.






<_<
>_>

...

tiny print: dil

Reply to Jarrett,

> "Bill Baxter" <dnewsgroup@billbaxter.com> wrote in message
> news:frce57$1vf5$1@digitalmars.com...
> 
>> Yes I believe Don has previously reported that CTFE manipulations on
>> strings basically *never* free the intermediate strings.  Basically
>> the CTFE engine needs a garbage collector (or equiv).
>> 
>> --bb
>> 
> You know, if we had a D compiler made in D, it'd be GCed.
> 

Odd, that's what the project the problem showed up on is.

My plan for CTFE is to compile all code to a high level IL byte code that can be converted to machine code for object files but can also be executed directly on some sort of VM, I should be able to shoehorn in a GC (or better) somewhere in that .

Bill Baxter wrote:
> BCS wrote:
>> I'm working on my parser generator and trying to shoe horn it into a smaller memory foot print. One approach I'm trying, is to replace recursive templates with CTFE function and string mixins. This is in an effort to reduce the number of types that get declared (mostly I'm using struct templates). The problem is that the CTFE functions seem to actually use MORE ram than the templates do. I've tried a number of things like converting to an O(n log n) concatenation system rather than a O(n^2) and other such tricks but everything seems to be going backwards. Any advice? (D v1 BTW) I can post code if anyone cares (it's kind of large: 1.2KLOC, takes a while to compile: ~6min, and use a lot of RAM: ~2GB).
>>
>> (after a bit more fiddling)
>>
>> I've got it down to 1GB of ram (Yeah!). My conclusions: don't build strings with CTFE unless it's a lot harder to do it with templates. It seems that CTFE is a bigger memory hog than templates even with temples memoizing everything into the symbol tables.
>>
>>
> 
> Yes I believe Don has previously reported that CTFE manipulations on strings basically *never* free the intermediate strings.  Basically the CTFE engine needs a garbage collector (or equiv).
> 
> --bb

grep for 'delete' in the front-end code. It's not there very much :-).

Robert Fraser wrote:
> grep for 'delete' in the front-end code. It's not there very much :-).

Maybe Walter should use the Boehm collector for the frontend?

Christopher Wright wrote:
> Robert Fraser wrote:
>> grep for 'delete' in the front-end code. It's not there very much :-).
> 
> Maybe Walter should use the Boehm collector for the frontend?

Doesn't he already do that? (He has often stated he uses a collector, and I'm not aware of any others for C++)

Frits van Bommel wrote:
> Christopher Wright wrote:
>> Robert Fraser wrote:
>>> grep for 'delete' in the front-end code. It's not there very much :-).
>>
>> Maybe Walter should use the Boehm collector for the frontend?
> 
> Doesn't he already do that? (He has often stated he uses a collector, and I'm not aware of any others for C++)

There's a mem.h that looks designed to potentially wrap a GC, but the actual implementation just calls system malloc and free.
Contains the comment:
/* This implementation of the storage allocator uses the standard C allocation package.
 */

That package's mem.malloc is used in constfold.c where a lot of the CTFE magic happens, so maybe wiring up mem.c to Boehm would fix it.  Looks like it wouldn't be too difficult to try out for someone who is able to build a working compiler.  So far that does not include me. :-(

--bb

Bill Baxter wrote:
> Frits van Bommel wrote:
>> Christopher Wright wrote:
>>> Robert Fraser wrote:
>>>> grep for 'delete' in the front-end code. It's not there very much :-).
>>>
>>> Maybe Walter should use the Boehm collector for the frontend?
>>
>> Doesn't he already do that? (He has often stated he uses a collector, and I'm not aware of any others for C++)
> 
> There's a mem.h that looks designed to potentially wrap a GC, but the actual implementation just calls system malloc and free.
> Contains the comment:
> /* This implementation of the storage allocator uses the standard C allocation package.
>  */
> 
> That package's mem.malloc is used in constfold.c where a lot of the CTFE magic happens, so maybe wiring up mem.c to Boehm would fix it.  Looks like it wouldn't be too difficult to try out for someone who is able to build a working compiler.  So far that does not include me. :-(

Oh, I guess that file (mem.c) was modified by Lindquist for LLVMDC (the derivative of DMDFE I'm most familiar with).
Anyone who wants to hook DMDFE up to Boehm can probably just use that version (<http://www.dsource.org/projects/llvmdc/browser/trunk/dmd/mem.c>).

Still, Walter claims to use a GC (see <http://www.digitalmars.com/d/1.0/faq.html#q7_3>). Though that doesn't specify he uses it in DMD itself, so maybe that's the source of the misunderstanding?