I have numerous problems with the program I wrote to test and benchmark D's
dynamic arrays.
The source code is attached. It compiles with D 0.79 and D 0.75(with minor
modification).

What it tries to do is to read file contents and fill dynamic array of
arrays(each of random length(1-20 bytes),
then reassemble the data back and write it to another file.

It seems to work Ok, but only on small(really small) files. On my system
under Win98 it
stops working with files bigger than 50kb. ;) Under XP it manages to work
with any size
I tried, but consuming catastrophic amounts of memory(up to 90mb per 800kb
file!) and time.
Of course program does nothing useful. ;) But it was intended to be
benchmark for memory
allocation and management speed.

Just out of curiousity I have implemented the same program in
Prolog(SWI-Prolog) and ran
it on the same files. Well the speed was just about the same and memory
consumption was
twice as small. This is unbelievable results, because prolog is fully
interpreted, linked-lists
are used instead of arrays and even file ops are done on element-by-element
basis
through tail recursion!

Probably my code is wrong somewhere or D's run-time is doing very bad job. Please prove me wrong, because I was(and still) going to use D extensively.

Alex A. B. wrote:
> I have numerous problems with the program I wrote to test and benchmark D's
> dynamic arrays.
> The source code is attached. It compiles with D 0.79 and D 0.75(with minor
> modification).
> 
> What it tries to do is to read file contents and fill dynamic array of
> arrays(each of random length(1-20 bytes),
> then reassemble the data back and write it to another file.
> 
> It seems to work Ok, but only on small(really small) files. On my system
> under Win98 it
> stops working with files bigger than 50kb. ;) Under XP it manages to work
> with any size
> I tried, but consuming catastrophic amounts of memory(up to 90mb per 800kb
> file!) and time.
> Of course program does nothing useful. ;) But it was intended to be
> benchmark for memory
> allocation and management speed.
> 
> Just out of curiousity I have implemented the same program in
> Prolog(SWI-Prolog) and ran
> it on the same files. Well the speed was just about the same and memory
> consumption was
> twice as small. This is unbelievable results, because prolog is fully
> interpreted, linked-lists
> are used instead of arrays and even file ops are done on element-by-element
> basis
> through tail recursion!
> 
> Probably my code is wrong somewhere or D's run-time is doing very bad job.
> Please prove me wrong, because I was(and still) going to use D extensively.
> 
> 

This is probably the memory allocation issue that been reported recently.  Here is the fix originally suggested by yaneurao.
In std.thread add foward declaration:

extern(Windows) export thread_hdl GetCurrentProcess();

In Thread.getCurrentThreadHandle() in std.thread change the line:

thread_hdl currentProcess = cast(thread_hdl)-1;
to
thread_hdl currentProcess = GetCurrentProcess();

Then rebuild phobos.lib  This fixed the problems I was encountering.

Paul

Anyone else know what is going on with this? Has someone tried it who
recompiled phobos? I'd like to know if the GC is really sucking up this much
memory.
thanks,
-Ben

"Paul Runde" <prunde@consolidated.net> wrote in message news:bvb1v6$1l8o$1@digitaldaemon.com...
> Alex A. B. wrote:
> > I have numerous problems with the program I wrote to test and benchmark
D's
> > dynamic arrays.
> > The source code is attached. It compiles with D 0.79 and D 0.75(with
minor
> > modification).
> >
> > What it tries to do is to read file contents and fill dynamic array of
> > arrays(each of random length(1-20 bytes),
> > then reassemble the data back and write it to another file.
> >
> > It seems to work Ok, but only on small(really small) files. On my system
> > under Win98 it
> > stops working with files bigger than 50kb. ;) Under XP it manages to
work
> > with any size
> > I tried, but consuming catastrophic amounts of memory(up to 90mb per
800kb
> > file!) and time.
> > Of course program does nothing useful. ;) But it was intended to be
> > benchmark for memory
> > allocation and management speed.
> >
> > Just out of curiousity I have implemented the same program in
> > Prolog(SWI-Prolog) and ran
> > it on the same files. Well the speed was just about the same and memory
> > consumption was
> > twice as small. This is unbelievable results, because prolog is fully
> > interpreted, linked-lists
> > are used instead of arrays and even file ops are done on
element-by-element
> > basis
> > through tail recursion!
> >
> > Probably my code is wrong somewhere or D's run-time is doing very bad
job.
> > Please prove me wrong, because I was(and still) going to use D
extensively.
> >
> >
>
> This is probably the memory allocation issue that been reported recently.  Here is the fix originally suggested by yaneurao. In std.thread add foward declaration:
>
> extern(Windows) export thread_hdl GetCurrentProcess();
>
> In Thread.getCurrentThreadHandle() in std.thread change the line:
>
> thread_hdl currentProcess = cast(thread_hdl)-1;
> to
> thread_hdl currentProcess = GetCurrentProcess();
>
> Then rebuild phobos.lib  This fixed the problems I was encountering.
>
> Paul

Ben Hinkle <bhinkle4@juno.com> ÓÏÏÂÝÉÌ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ:bvbsbv$90$1@digitaldaemon.com...
> Anyone else know what is going on with this? Has someone tried it who recompiled phobos? I'd like to know if the GC is really sucking up this
much
> memory.
> thanks,
> -Ben
Yes, I have recompiled phobos, while memory
allocation problem with Windows 98 was gone,
it still does consume HUGE amounts of memory.

Even SWI-Prologs' GC does a much better job
at the same task.

Maybe it's time to consider that collecting background thread?

"Alex A. B." <freshmind@fromru.com> wrote in message news:bvc9eo$lj8$1@digitaldaemon.com...
>
> Ben Hinkle <bhinkle4@juno.com> ÓÏÏÂÝÉÌ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ:bvbsbv$90$1@digitaldaemon.com...
> > Anyone else know what is going on with this? Has someone tried it who recompiled phobos? I'd like to know if the GC is really sucking up this
> much
> > memory.
> > thanks,
> > -Ben
> Yes, I have recompiled phobos, while memory
> allocation problem with Windows 98 was gone,
> it still does consume HUGE amounts of memory.
>
> Even SWI-Prologs' GC does a much better job
> at the same task.
>
>
>
>

Glancing over the test code it doesn't look like it is doing much
collecting, only lots and lots of small allocation. There are some reallocs
being called by ~= but in general it looks like there are small (1 to 20
bytes) chunks and few 800K chunks (for an 800K input). Is it overhead on
small objects? Is it poor recylcing of realloced arrays? I notice the last
loop as it catenates all the small chunks back together the GC never has to
alloc a small chunk - it is always resizing the output array to something
larger so if the old array was just put on the free list it would never get
reused).
Does the GC ever merge free'd blocks? I can't tell from glancing at the
gcx.d what is causing this code to perform like it is. It would be a pity to
get D 1.0 and have it dismissed because of GC corner cases that make it look
bad.

-Ben

"Matthew" <matthew.hat@stlsoft.dot.org> wrote in message news:bvcaoe$nri$1@digitaldaemon.com...
> Maybe it's time to consider that collecting background thread?
>
> "Alex A. B." <freshmind@fromru.com> wrote in message news:bvc9eo$lj8$1@digitaldaemon.com...
> >
> > Ben Hinkle <bhinkle4@juno.com> ÓÏÏÂÝÉÌ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ:bvbsbv$90$1@digitaldaemon.com...
> > > Anyone else know what is going on with this? Has someone tried it who recompiled phobos? I'd like to know if the GC is really sucking up
this
> > much
> > > memory.
> > > thanks,
> > > -Ben
> > Yes, I have recompiled phobos, while memory
> > allocation problem with Windows 98 was gone,
> > it still does consume HUGE amounts of memory.
> >
> > Even SWI-Prologs' GC does a much better job
> > at the same task.
> >
> >
> >
> >
>
>

"Alex A. B." <freshmind@fromru.com> wrote in message news:bvc9eo$lj8$1@digitaldaemon.com...
>
> Ben Hinkle <bhinkle4@juno.com> ÓÏÏÂÝÉÌ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ:bvbsbv$90$1@digitaldaemon.com...
> > Anyone else know what is going on with this? Has someone tried it who recompiled phobos? I'd like to know if the GC is really sucking up this
> much
> > memory.
> > thanks,
> > -Ben
> Yes, I have recompiled phobos, while memory
> allocation problem with Windows 98 was gone,
> it still does consume HUGE amounts of memory.
>
> Even SWI-Prologs' GC does a much better job
> at the same task.

I've attached a small program to ferret out some GC behaviors. Playing
around you'll see that
1) the smallest allocable chunks come in sizes 16, 32, 64 etc bytes
2) resizing an array dynamically one element at a time is very inefficient.

I think these two factors contribute to the memory usage you saw with your test program. I haven't tried modifying your code exactly, but see what happens when you replace the "rand()%20+1" in your program with numbers around 16 or 32. Also try getting rid of the "~=" and replace it with explicitly setting the length so some large numbers. At one point Walter recommended setting the length to a large number and then back to zero in order to "preallocate" space. This didn't completely fix the problem but it helped a little.

-Ben