On 05.09.2012 00:07, Andrei Alexandrescu wrote:
> On 9/4/12 9:38 PM, Rainer Schuetze wrote:
>> IIRC it was removed because compile times more than doubled.
>
> That always struck me as a simple parameter choosing problem. Can't we
> set the limit at which the GC enters in action high enough so only the
> few programs that really need a lot of RAM actually incur a collection?
> What am I missing?
>
>> Regarding the bail out with less than 1 GB allocated I suspect that the
>> dmc runtime uses VirtualAlloc with sizes below the allocation
>> granularity (which is 64kB on 32-bit windows). This waists virtual
>> address space, so another allocation fails even if the actually
>> allocated physical memory is well below the 2GB limit.
>>
>> The MS-cl-compiled version of dmd does not exhibit this problem (and
>> compiles D code considerably faster).
>
> Interesting. Can we fix that?

If I read the dmc runtime library sources correctly (heap32/heap.cpp), heap allocations work with a minimum chunk size of 64 kB but unfortunately 4 bytes get added to the requested size before calling VirtualAlloc. That actually blocks 128kB of virtual address space, but only 64k+4 bytes are used.

So, I recommend just aligning to the next multiple of 64kB in RTLHeap::MoreCore. Walter, can you rebuild snn.lib with an appropriate fix?

_______________________________________________
phobos mailing list
phobos@puremagic.com
http://lists.puremagic.com/mailman/listinfo/phobos

2012/9/5 David Simcha <dsimcha@gmail.com>:
> I think we need separate compilation for non-unittest Phobos compiles, too.

I've proposed it in the past, but Walter had been against it. https://github.com/D-Programming-Language/phobos/pull/280#issuecomment-2259177

> Antti-Ville's precise heap scanning, which will hopefully be integrated soon (I've been meaning to ping him on that) makes the non-unittest Phobos build process run out of memory on Windows because of all the template instantiations necessary.  Alternatively, we could bring back the compile time GC.  Walter, why was compile-time GC removed again?

Compile-time GC had broke some pull requests, at least including mine. https://github.com/D-Programming-Language/dmd/pull/147#issuecomment-2282965

And, as far as I know, it had increased compilation time.

Adding gc in Windows: https://github.com/D-Programming-Language/dmd/commit/2f7ce3475

Removing gc: https://github.com/D-Programming-Language/dmd/commit/8515995

I couldn't find the commit for stopping gc use...

Kenji Hara
_______________________________________________
phobos mailing list
phobos@puremagic.com
http://lists.puremagic.com/mailman/listinfo/phobos

On 05.09.2012 00:39, Rainer Schuetze wrote:
> On 05.09.2012 00:07, Andrei Alexandrescu wrote:
>> Interesting. Can we fix that?
>
> If I read the dmc runtime library sources correctly (heap32/heap.cpp),
> heap allocations work with a minimum chunk size of 64 kB but
> unfortunately 4 bytes get added to the requested size before calling
> VirtualAlloc. That actually blocks 128kB of virtual address space, but
> only 64k+4 bytes are used.
>
> So, I recommend just aligning to the next multiple of 64kB in
> RTLHeap::MoreCore. Walter, can you rebuild snn.lib with an appropriate fix?

I patched snn.lib to that effect, i.e. replaced this code

  if (Size < 0x10000)
    Size = 0x10000;
  else
    Size += Size/5;  // Allocate 20% more to eliminate fragmentation

with this:
  Size = ((Size + 4 + Size/5 + 0xffff) & ~0xffff) - 4;
  // 4 is added in call to sbrk

and ran this little test program:

import core.stdc.stdlib;
import core.stdc.stdio;

void main()
{
	for(int i = 0; ; i++)
		if(malloc(1024) is null)
		{
			printf("allocated %d MB\n", i/1024);
			break;
		}
}

Compiling with the original snn.lib and running under Win8/64bit, it outputs 962 MB of allocated memory, 1954 MB if I set the large-address-aware bit.

With the patch, it can allocate 1925 MB and 3906 MB, respectively.

It takes minutes to run this little program (no swapping involved, I have 8GB of memory), slowing down to allocating about 5MB/s when watching progress in the process explorer. That seems to suggest that the heap function do not deal well with that amount of memory or at least with this allocation pattern.

If you want to try it yourself, start your favorite hex editor, load snn.lib, find

81 FE 00 00 01 00 73 07 BE 00 00 01 00 EB 0D 8B C6 BB 05 00 00 00 31 D2 F7 F3 01 C6 8D 56 04

and replace it with

89 f0 BB 05 00 00 00 31 D2 F7 F3 8D 84 30 03 00 01 00 25 00 00 FF FF 8D 70 FC 90 90 8D 56 04

Rainer

_______________________________________________
phobos mailing list
phobos@puremagic.com
http://lists.puremagic.com/mailman/listinfo/phobos