On Thu, 09 May 2013 22:14:41 -0400, Ali Çehreli <acehreli@yahoo.com> wrote:

> On a 32-bit system string is 8 bytes (e.g. by pragma(msg, string.sizeof)). So you have a single array of 800 million bytes. Then, each of those strings point at 22 bytes of individully allocated memory areas.

No, each array points at static data.  Strings are immutables stored in the data segment.

> Note that 800 million is about 20% of the entire addressable memory of that system. If you have a single value that happens to look like a pointer into that memory, that huge block will remain in memory forever. The funny thing about dmd's current conservative GC is that even an integer can be mistaken to be a pointer into that memory.
>
> So that's the problem today.

Correct, the only solution at the moment is to deallocate explicitly.

It is important to keep in mind that C# has a *precise* garbage collector, D has a *conservative* garbage collector.  C# will always do better than D at GC as long as this is true.

64 bit address space also helps tremendously.

> I don't know the exact requirement here but if you really must have 100 million strings at one time, then you may want to do your own memory management. The following should work:
>
> import std.stdio;
> import core.memory;
>
> void main()
> {
>      enum size_t times = 10;
>      enum size_t stringCount = 100_000_000;
>
>      foreach (i; 0 .. times) {
>          auto rawMemory = cast(string*)GC.calloc(string.sizeof * stringCount);
>
>          // D's cool feature of making a slice from raw pointer
>          auto tempArray = rawMemory[0 .. stringCount];

You can allocate normally, and just use free on the pointer

>
>          foreach (ref s; tempArray) {
>              s = "aaaaaaaaaaaaaaaaaaaaa";
>          }
>
>          GC.free(rawMemory);

GC.free(tempArray.ptr); // also works

>          // You may want to set tempArray to null at this point to prevent
>          // using it accidentally but it is not necessary.
>
>          writefln("Done with allocation %s/%s; please press Enter",
>                   i + 1, times);
>          readln();
>      }
> }

-Steve

On 05/09/2013 07:43 PM, Steven Schveighoffer wrote:

> On Thu, 09 May 2013 22:14:41 -0400, Ali Çehreli <acehreli@yahoo.com> wrote:

>> Then, each of those strings point at 22 bytes of individully allocated
>> memory areas.
>
> No, each array points at static data.  Strings are immutables stored in
> the data segment.

You are right. They start at pointing to the same string but as soon as each string is used in a special way, say each gets appended a char, then they own their special characters:

            s = "aaaaaaaaaaaaaaaaaaaaa";

            // Some time later:
            s ~= 'x';

Now each s.ptr is different.

Ali

Thanks for the explanation guys.
Hope there will be improvements in D's GC in the future.

Also "delete temp_array" also works instead of "GC.free(temp_array.ptr)".
And you also need to call "GC.minimize()" after a delete/GC.free() if you want to shrink the memory use after the for loop:

        temp_array = new string[100000000];
	for(int i=0;i<100000000;i++)
	{
		temp_array[i] = "aaaaaaaaaaaaaaaaaaaaa";
	}
	delete temp_array; or GC.free(temp_array.ptr);
	temp_array = null;
	GC.minimize();  //releases the memory, works without this line

Is "delete" safe for this kind of use?

Thanks guys

On Fri, 10 May 2013 04:44:14 -0400, Matic Kukovec <matic.kukovec@pametnidom.si> wrote:

> Thanks for the explanation guys.
> Hope there will be improvements in D's GC in the future.
>
> Also "delete temp_array" also works instead of "GC.free(temp_array.ptr)".
> And you also need to call "GC.minimize()" after a delete/GC.free() if you want to shrink the memory use after the for loop:
>
>          temp_array = new string[100000000];
> 	for(int i=0;i<100000000;i++)
> 	{
> 		temp_array[i] = "aaaaaaaaaaaaaaaaaaaaa";
> 	}
> 	delete temp_array; or GC.free(temp_array.ptr);
> 	temp_array = null;
> 	GC.minimize();  //releases the memory, works without this line
>
> Is "delete" safe for this kind of use?

Delete is unsafe.  It is being deprecated actually (not sure when).  Becuase it cannot zero out all pointers that reference that data, you can have dangling pointers to the data.

delete is replaced by the function destroy, which finalizes data, and GC.free to deallocate.  Only the GC.free part is unsafe, destroy is OK to use.

However, for arrays, destroy is equivalent to setting the value to null.

-Steve