I am very bad at naming things.

This is a module containing classes to manage raw data in external memory. It provides semantics similar to built-in void[] arrays, but has the following advantages:

* Faster allocation and deallocation, since memory is requested from the OS directly as whole pages
* Greatly reduced chance of memory leaks due to stray pointers
* Overall improved GC performance due to reduced size of managed heap
* Memory is immediately returned to the OS when data is deallocated

I've been using it in my own programs, and it saved me from many headaches.
Personally, I think that it is extremely useful for any applications that deal with non-trivial amounts of raw data - web/file servers, multimedia applications, video games, etc., and belongs in the standard library.

Currently D1/Phobos, but should be easy to port to D2 or Tango.
I believe it can be made memory-safe by removing deleteContents and fixing the append clobber issue.

Source and more info here: http://github.com/CyberShadow/data.d

If you see anything that can be improved, feel free to fork the github repo, or post patches, or just comment on it.

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

"Vladimir Panteleev" <vladimir@thecybershadow.net> wrote in message news:op.vfvdtwgwtuzx1w@89-28-59-99.starnet.md...
>
> If you see anything that can be improved, feel free to fork the github repo, or post patches, or just comment on it.

VirtualAlloc returns chunks that have 'dwAllocationGranularity' granularity, which is 64K on every Windows OS I've used. So allocating a page, 4K, will actualy get you 64K.

So using VirtualAlloc as a replacement for malloc is very wasteful unless the allocations are larger that 64K.

You might want to look at HeapCreate and it's siblings. (on windows anyway)

On Thu, 15 Jul 2010 11:03:57 +0300, JimBob <jim@bob.com> wrote:

> VirtualAlloc returns chunks that have 'dwAllocationGranularity' granularity,
> which is 64K on every Windows OS I've used. So allocating a page, 4K, will
> actualy get you 64K.
>
> So using VirtualAlloc as a replacement for malloc is very wasteful unless
> the allocations are larger that 64K.
>
> You might want to look at HeapCreate and it's siblings. (on windows anyway)

dwAllocationGranularity only controls the granularity of the allocation address, but not of the allocation size. A simple program that calls VirtualAlloc 16384 times uses up 1 GB of the address space, but only 64 MB of actual memory. So, while it is a waste of address space, it's not a waste of system memory.

I think that using heaps of any kind defeats part of this module's purpose, for the reason that due to heap fragmentation it is not possible to return memory to the OS directly (which is only possible to do in whole pages). There is also a small performance cost involved with using heaps.

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

"Vladimir Panteleev" <vladimir@thecybershadow.net> wrote in message news:op.vfvnf1bvtuzx1w@89-28-59-99.starnet.md...
> On Thu, 15 Jul 2010 11:03:57 +0300, JimBob <jim@bob.com> wrote:
>
>> VirtualAlloc returns chunks that have 'dwAllocationGranularity'
>> granularity,
>> which is 64K on every Windows OS I've used. So allocating a page, 4K,
>> will
>> actualy get you 64K.
>>
>> So using VirtualAlloc as a replacement for malloc is very wasteful unless the allocations are larger that 64K.
>>
>> You might want to look at HeapCreate and it's siblings. (on windows anyway)
>
> dwAllocationGranularity only controls the granularity of the allocation address, but not of the allocation size. A simple program that calls VirtualAlloc 16384 times uses up 1 GB of the address space, but only 64 MB of actual memory. So, while it is a waste of address space, it's not a waste of system memory.

Appologies, msdn says you're right.

Which sucks cause I recently rewrote a bunch of code because of that misunderstanding.

Thanks.

Vladimir Panteleev wrote:
> On Thu, 15 Jul 2010 11:03:57 +0300, JimBob <jim@bob.com> wrote:
> 
>> VirtualAlloc returns chunks that have 'dwAllocationGranularity' granularity,
>> which is 64K on every Windows OS I've used. So allocating a page, 4K, will
>> actualy get you 64K.
>>
>> So using VirtualAlloc as a replacement for malloc is very wasteful unless
>> the allocations are larger that 64K.
>>
>> You might want to look at HeapCreate and it's siblings. (on windows anyway)
> 
> dwAllocationGranularity only controls the granularity of the allocation address, but not of the allocation size. A simple program that calls VirtualAlloc 16384 times uses up 1 GB of the address space, but only 64 MB of actual memory. So, while it is a waste of address space, it's not a waste of system memory.

In a 32-bit process, the waste of address space is sometimes more critical nowadays. There is only 2GB virtual memory available (3GB with some tweaks), and with the example above, VirtualAlloc fails well before allocating 128 MB.

== Quote from Rainer Schuetze (r.sagitario@gmx.de)'s article
> Vladimir Panteleev wrote:
> > On Thu, 15 Jul 2010 11:03:57 +0300, JimBob <jim@bob.com> wrote:
> >
> >> VirtualAlloc returns chunks that have 'dwAllocationGranularity'
> >> granularity,
> >> which is 64K on every Windows OS I've used. So allocating a page, 4K,
> >> will
> >> actualy get you 64K.
> >>
> >> So using VirtualAlloc as a replacement for malloc is very wasteful unless the allocations are larger that 64K.
> >>
> >> You might want to look at HeapCreate and it's siblings. (on windows
> >> anyway)
> >
> > dwAllocationGranularity only controls the granularity of the allocation address, but not of the allocation size. A simple program that calls VirtualAlloc 16384 times uses up 1 GB of the address space, but only 64 MB of actual memory. So, while it is a waste of address space, it's not a waste of system memory.
> In a 32-bit process, the waste of address space is sometimes more critical nowadays. There is only 2GB virtual memory available (3GB with some tweaks), and with the example above, VirtualAlloc fails well before allocating 128 MB.

Yea, but I wonder how much longer it is going to be before 32-bit is dead as a dodo except on things like netbooks.  Frankly, it's about time for it to die, because dealing w/ address space limitations when plenty of physical memory is available just plain sucks.  DMD is in the process of being ported to 64-bit as we speak.  Almost any Linux distro is available in 64-bit.  IIRC (I can't remember for sure where I heard this) Windows 7 will be the last 32-bit version, and it seems like most people who have migrated use the 64-bit version.

On the hardware end, 64-bit is now 6-7 years old, which has to be a standard deviation or two older than the average age at which computers get replaced.

On Fri, 16 Jul 2010 01:58:27 +0300, dsimcha <dsimcha@yahoo.com> wrote:

> On the hardware end, 64-bit is now 6-7 years old, which has to be a standard  deviation or two older than the average age at which computers get replaced.

FWIW, AFAIK data.d is 64-bit ready :D

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

On Fri, 16 Jul 2010 01:20:07 +0300, Rainer Schuetze <r.sagitario@gmx.de> wrote:

> There is only 2GB virtual memory available (3GB with some tweaks)

The allocation granularity doesn't affect virtual memory either (at least according to all Process Explorer indications).

Does this 2 or 3GB limitation only affect 32-bit operating systems? On my 64-bit Windows, with /LARGEADDRESSAWARE, a simple program can do close to 64K (65062 for me) 1-byte VirtualAllocs.

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

On Fri, 16 Jul 2010 02:20:16 +0300, Vladimir Panteleev <vladimir@thecybershadow.net> wrote:

> Does this 2 or 3GB limitation only affect 32-bit operating systems? On my 64-bit Windows, with /LARGEADDRESSAWARE, a simple program can do close to 64K (65062 for me) 1-byte VirtualAllocs.

Said simple program in case anyone wants to test, excuse the C:

#include <windows.h>
#include <stdio.h>

#define N (64*1024)

void main()
{
	int i;
	char s[10];

	gets(s);
	for (i=0; i<N; i++)
	{
		void *p = VirtualAlloc(NULL, 1, MEM_COMMIT, PAGE_READWRITE);
		if (p==NULL)
		{
			printf("Allocation failed at %d\n", i);
			break;
		}
		*(char*)p = 17;
		if (i==0 || i==N-1 || p==NULL)
			printf("%p\n", p);
	}
	gets(s);
}

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

"dsimcha" <dsimcha@yahoo.com> wrote in message news:i1o3qj$13b4$1@digitalmars.com...
>
> Yea, but I wonder how much longer it is going to be before 32-bit is dead
> as a
> dodo except on things like netbooks.  Frankly, it's about time for it to
> die,
> because dealing w/ address space limitations when plenty of physical
> memory is
> available just plain sucks.

I have 1GB. (And I get by just fine.)