On Friday, 7 April 2017 at 21:02:33 UTC, ketmar wrote:
> Nierjerson wrote:
>
>> How to implement trick is this and are you 100% sure it works?
>>
>> e.g.,
>>
>> char[] x;
>>
>> x.length = 65536;
>> x.length = 0;
>
> this won't work. the moment you did `.length = 0;`, you are returned to point zero.
>
> what you have to do is to maintain "shadow length" yourself, like this:
>
> 	x.length = 65536;
> 	size_t xpos = 0;
> 	
> 	void put (const(char)[] s...) {
> 		foreach (immutable char ch; s) {
> 			if (xpos == x.length) x.length += 65536; // or anything
> 			x[xpos++] = ch;
> 		}
> 	}

thanks, I'll try it... seems like it is basically appender though?

Jethro wrote:

> On Friday, 7 April 2017 at 21:02:33 UTC, ketmar wrote:
>> Nierjerson wrote:
>>
>>> How to implement trick is this and are you 100% sure it works?
>>>
>>> e.g.,
>>>
>>> char[] x;
>>>
>>> x.length = 65536;
>>> x.length = 0;
>>
>> this won't work. the moment you did `.length = 0;`, you are returned to point zero.
>>
>> what you have to do is to maintain "shadow length" yourself, like this:
>>
>> 	x.length = 65536;
>> 	size_t xpos = 0;
>> 	
>> 	void put (const(char)[] s...) {
>> 		foreach (immutable char ch; s) {
>> 			if (xpos == x.length) x.length += 65536; // or anything
>> 			x[xpos++] = ch;
>> 		}
>> 	}
>
> thanks, I'll try it... seems like it is basically appender though?

yeah. but the key here is to not use any fancy data structures, it is important. the more intermediaries you have, the less control over copying is left for you. also, no slice assigns too -- it is dangerous, as it can easy go out of control.

On Thursday, 6 April 2017 at 20:49:00 UTC, Nierjerson wrote:
> I am running out of memory trying to generate CTFE code. It is quite large generation but just repetitive loop acting on an array item.
>
> Surely 16GB should be enough to compile such a thing? I am using 64-bit dmd. It climes to about 12GB then eventually drops down to around 1GB and then back up to 16GB and then quits.
>
> I cannot generate the array in parts and stick them all together because they are all interdependent(but not much, nothing that should actually cost memory).
>
> Seems that dmd not freeing up memory for speed is gonna start causing problems with complex template generation.
>
> Is there any way to fix this? A special switch that will enable the compiler to reduce memory consumption(free unused stuff) or use the swap file?
>
> https://github.com/IllusionSoftware/COM2D/
>
> At the very least have something to give feedback on how to reduce memory consumption. Leaving things up in the air for programmers to stumble upon after a lot of work is not good.
>
> On the "Error: Out of Memory" at least report some statistics on functions and lines and how much memory they have used and how many times they have been called.

Some years ago I managed to force DMD to collect memory during CTFE by loading the malloc replacement of Boehm GC using LD_PRELOAD on Linux. It sounds totally crazy, but it worked...

Check the last part called "Simplified leak detection under Linux" of this link:

https://www.hboehm.info/gc/leak.html

You can ignore the leak detection aspect and just build and preload libgc.so. It will (very conservatively) collect memory if a malloc fails. Lets hope DMD drops old references during CTFE...

Please report back if this still works.