class Test
{
    int[10] values;
}


void main()
{
    Test[1000_000] arr;
    printf( "sizeof Test : %d\n\x00", Test.classinfo.init.length );
    printf( "sizeof Array: %d\n\x00", arr.sizeof );
    for( int j = 0; j < 1000; j++ )
    {
        for( int i = 0; i < 1000; i++ )
        {
            arr[j*1000+i] = new Test;
        }
        printf( "loop %d\n\x00", j );
    }
}

The program runs and gets slower and slower.
After 8min at loop 700 I canceled.
Is this speed acceptable for allocating 48MB (+4MB for the array)?

I tried to use a XML DOM parser, parsing 1,5MB xml. This seems to take endless.

Is there a workaround? e.g. preallocating memory in one piece.

Frank Benoit wrote:
> class Test
> {
>     int[10] values;
> }
> 
> 
> void main()
> {
>     Test[1000_000] arr;
>     printf( "sizeof Test : %d\n\x00", Test.classinfo.init.length );
>     printf( "sizeof Array: %d\n\x00", arr.sizeof );
>     for( int j = 0; j < 1000; j++ )
>     {
>         for( int i = 0; i < 1000; i++ )
>         {
>             arr[j*1000+i] = new Test;
>         }
>         printf( "loop %d\n\x00", j );
>     }
> }
> 
> The program runs and gets slower and slower.
> After 8min at loop 700 I canceled.
> Is this speed acceptable for allocating 48MB (+4MB for the array)?
> 
> I tried to use a XML DOM parser, parsing 1,5MB xml. This seems to take
> endless.
> 
> Is there a workaround? e.g. preallocating memory in one piece.

I used the following re-write of your program:

# private import std.stdio;
# private import cashew.utils.timer;
#
# class Test {
#   int[10] values ;
# }
#
# void main () {
#   Test[100_000] arr;
#   writefln(r"sizeof Test : %d"c, Test.classinfo.init.length);
#   writefln(r"sizeof Array: %d"c, arr.sizeof);
#   { auto Timer t0 = new Timer(r"loop"c);
#   foreach(i, inout x; arr) {
#     x = new Test;
#     if (((i + 1) % 100) == 0)
#       writefln(r"loop %d"c, i + 1);
#   }
#   }//timer
# }

Do note I dropped the size of 'arr' from 1000000 to 100000.  This has nothing to do with with the issue, so far as I know, and everything to do with a stack size problem on the machine I'm sitting at right now.  Don't ask... pos.

Anyhow, the Timer reports from about thirty or so runs of that program spanned between 0.66 seconds and 0.77 seconds total, so I shouldn't expect your original to take more than a second or two.

-- Chris Nicholson-Sauls

To see the difference to Java...

public class Alloc {

	int[] val;
	public Alloc()
	{
		val = new int[10];
	}

	public static void main(String[] args) {
		Alloc[] arr = new Alloc[1000000];
		for( int j = 0; j < 1000; j++ )
		{
	        	for( int i = 0; i < 1000; i++ )
	        	{
	            		arr[j*1000+i] = new Alloc();
	        	}
	        	System.out.println( "loop "+j );
	    	}
		System.out.println( "complete " );
	}
}


Note, this Java program make a second allocation for the array and does also initialize the memory.

The program was finished after ONE second.
This 1000 times faster like the D program.

So I have to say that again... The GC is a show stopper and need to be changed.

I am completely sorry.
I used a wrong verson of libphobos.
With the original one it works fast.

Frank

Frank Benoit wrote:
> 	int[] val;
> 	public Alloc()
> 	{
> 		val = new int[10];
> 	}

Just for the record, you can use the same construct above in D.  In fact, you could technically do such silliness as:
# auto val = new int[10];

Madness.  But occasionally useful madness.

-- Chris Nicholson-Sauls

Frank Benoit schrieb:
> I am completely sorry.
> I used a wrong verson of libphobos.
> With the original one it works fast.
> 
> Frank

BTW: The only difference was in the makefiles:

original: DFLAGS = -O -inline -release
My one  : DFLAGS = -g -debug

With the original options and an own compiled libphobos.a it is also completelly OK. ~500ms

On Mon, 27 Mar 2006 21:05:55 +0200, Frank Benoit wrote:

> class Test
> {
>     int[10] values;
> }
> 
> void main()
> {
>     Test[1000_000] arr;
>     printf( "sizeof Test : %d\n\x00", Test.classinfo.init.length );
>     printf( "sizeof Array: %d\n\x00", arr.sizeof );
>     for( int j = 0; j < 1000; j++ )
>     {
>         for( int i = 0; i < 1000; i++ )
>         {
>             arr[j*1000+i] = new Test;
>         }
>         printf( "loop %d\n\x00", j );
>     }
> }
> 
> The program runs and gets slower and slower.
> After 8min at loop 700 I canceled.
> Is this speed acceptable for allocating 48MB (+4MB for the array)?
> 
> I tried to use a XML DOM parser, parsing 1,5MB xml. This seems to take endless.
> 
> Is there a workaround? e.g. preallocating memory in one piece.

I made a slight change to the code and it worked well.

I moved 'Test[1000_000] arr;' to the module level so it wouldn't be stack allocated, and I changed it to 'Test[1000_000] arr = void;' to avoid preinitializing it.

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Down with mediocracy!"
28/03/2006 10:30:30 AM

"Frank Benoit" <frank@nix.de> wrote in message news:e09ju8$28nc$1@digitaldaemon.com...
> Frank Benoit schrieb:
>> I am completely sorry.
>> I used a wrong verson of libphobos.
>> With the original one it works fast.
>>
>> Frank
>
> BTW: The only difference was in the makefiles:
>
> original: DFLAGS = -O -inline -release
> My one  : DFLAGS = -g -debug
>
> With the original options and an own compiled libphobos.a it is also completelly OK. ~500ms

By the way:
SGML tokenizer (aka HTML/XML pull parser) in Harmonia is
capable to scan 30mb XML file in one second. It depends on hardware but
order of magnitude is like this.
It does not allocate any memory while parsing.

Andrew.

Andrew Fedoniouk wrote:
> By the way:
> SGML tokenizer (aka HTML/XML pull parser) in Harmonia is
> capable to scan 30mb XML file in one second. It depends on hardware but
> order of magnitude is like this.
> It does not allocate any memory while parsing.
> 
> Andrew. 
> 
> 
Just looked at it briefly and I noticed that Scanner seems to only take in a string. Do you plan on adding a file stream as input?

Zz

"ZZ" <ZZ@zz.com> wrote in message news:e0ba8l$20k5$1@digitaldaemon.com...
> Andrew Fedoniouk wrote:
>> By the way:
>> SGML tokenizer (aka HTML/XML pull parser) in Harmonia is
>> capable to scan 30mb XML file in one second. It depends on hardware but
>> order of magnitude is like this.
>> It does not allocate any memory while parsing.
>>
>> Andrew.
> Just looked at it briefly and I noticed that Scanner seems to only take in a string. Do you plan on adding a file stream as input?

If you need to scan file then you can use MMFile.
module harmonia.io.mmfile;
( http://harmonia.terrainformatica.com )

or the one from phobos.

For stream parsing you need to create your own scanner and override

StreamScanner:Scanner
{
   get_char()
   switch_input()
   currentFragment() (optional, error reporting)
}

>
> Zz