In article <ctohtt$10mr$1@digitaldaemon.com>, Manfred Nowak says...
>[...]
>> large blocks of data (dynamic arrays are not meant for that, never
>> were).
>
>From where do you have this wisdom? If so please explain the prereqiesites for the usage of dynamic arrays and for fixed arrays as well.
>

Possibly the most compelling fact, that D's arrays are not for huge chunks of memory, is that they rely on copy-on-write semantics.

With a 1GB chunk of memory in a single D array, modifications to slices and concatenations will would result in a complete realloc and copy; so you could only use *one-half to a third* of your system's memory... and that's on a good day with agressive memory management and no GC.  Also, it's ill-advised to use a single array directly for something like this simply due to the cache-misses that are likely to result from random-accesses on a 1GB structure; the same goes for virtually every language out there.

To that end D sits frimly in the "trade more space for less running time" optimization camp, which is fine for the majority of tasks out there. Superscale blobs of data require a behavior that certainly can be expressed in D, but is not enshrined in it's underlying design.

IMO, the optimal (all-round) solution for working with massive data structures, would approach the complexity of a memory manager and not a series of simple array manipulations.  From there, you could implement an array-like interface to make it more friendly to use, but it would still be a far cry from a true array.


- EricAnderton at yahoo

Anders F Björklund wrote:

> Dawid Ciężarkiewicz wrote:
> 
>> Your post just remind me that case. I'm going even further. Why to initialize anything at all? In 99.9% cases variables are initialized one or two lines after creation and default isn't used at all.
> 
> It's for the other 0.1% percent, where forgetting to initialize a variable causes a subtle bug ? In other languages, such as Objective-C for instance, these are separate events altogether:

I can agree that setting floats to NAN make sense, but setting forgotten int to arbitrary value won't help program so much. It will let him give same errors rather than random ones. :)


> PreInitializing and GarbageCollecting are a whole lot easier to use.

I agree that GrabageCollecting is *necessary* for modern computer programing language. And I agree that should be a way to disable it if needed. Just as with variable initialization - sometimes (in critical parts of code) it would be good to could disable this.

> And for local variables, a reasonably good compiler should be able to optimize out the .init value, if it's just replaced right away...

This is part of the answer that I expected and I'm glad to hear, but what about variables that are not expected to have any value for start and are initialized later?

-- 
Dawid Ciężarkiewicz | arael
jid: arael@fov.pl

In article <ctoitr$11pk$1@digitaldaemon.com>, Norbert Nemec says...
>Any ideas for a possible syntax
>specifying "This variable should not be initialized"? It should be possible
>both for variables as well as for dynamically allocated memory or class
>members.

This sounds like a job for a compiler pragma.

> pragma(noinit) int a; // a gets a random value now (just like C!).
> int[] b;
> pragma(noinit){
>     b = new int[1024*1024*1024]  // b gets an uninitalized 1GB block.
> }

The nice thing about this is that pragmas in D are not be ignored if not understood.  So while compiler dependent, the code won't compile on D compilers that don't support 'noinit'.

- EricAnderton at yahoo

Norbert Nemec wrote:

> For C - of course, one should be able to bind in existing C code, but in the
> long term, there should not be any reason to write new code in C if you
> have a D compiler at hand.

I see D as a nice replacement for C++ in the (very) long run,
but I will continue to use either C or Java when they fit better...


But I agree that D is a bit strange in that it's pretty easy to
e.g. dereference null, but hard to e.g. allocate uninited memory ?

So far the performance has been good (just a few Mac OS X quirks still),
and seems to be one of the key points of D. So it's right to address it.


Maybe the auto initialization can be be replaced with an error if you
try to actually use the value without setting it first ? Like in Java.

(Java does D-style init of members, but only such errors for local vars.
 Not sure how much work it is for the compiler to catch such errors ?)

--anders

Dawid Ciężarkiewicz wrote:

> This is part of the answer that I expected and I'm glad to hear, but what
> about variables that are not expected to have any value for start and are
> initialized later?

Hard to say in the general case, I recommend playing with -O and disasm?
(if you use GDC, you can compare -O0...-O3, and get asm output with -S)

--anders

In article <ctokln$13jb$1@digitaldaemon.com>, Ben Hinkle says...
>
>
>"Manfred Nowak" <svv1999@hotmail.com> wrote in message news:ctmtne$2car$1@digitaldaemon.com...
>>I wrote about this before.
>>
>> There is a well known time/space-tradeoff for the preInitialization of arrays: using about three times the space one can lazy initialize an array.
>>
>> But this technic is useless within D, because of the automaatic preInitialization, which currently eats up about 3 cycles per byte.
>>
>> Please awaken to, that on a 3GHz machine the busy preInitalization of one GB then lasts one second. And the coming 64-bit-machine will have up to some TB of main memory. Current mainboards can already hold up to 4GB, which ist the current main memory limit for win32.
>>
>> Check again, that to preInitialize one TB you have to wait more than 15 minutes only to wait at least another 15 minutes until your videoediting can start, if no precautions are taken.
>>
>> We need an option to switch automatic preInitialization of arrays off.
>>
>> -manfred
>
>The ironic part is that the GC itself has malloc but the D interface to it always clears the result. See src/phobox/internal/gc.d routine _d_newarrayi. It would be really nice to have the following added to the GC interface:
>  void* malloc(size_t len) { return _gc.malloc(len); }
>Ah, to have something so close and yet so far away...
>
>And while I'm at it how about exposing _gc.realloc, _gc.free and _gc.capacity, too. oh, now I'm just dreaming I know.
>
>-Ben

As look as we're doing wishful thinking, maybe we could just put a readable/writeable property in the GC module called: "noinit". After _d_newarrayi malloc's the memory, it checks "noinit" to see if it should initialize or not. The majority of programmers would never touch this setting. Those that have to clear 1 TB of memory have the option.

jcc7

> "Manfred Nowak" <svv1999@hotmail.com> wrote in message 
>>But this technic is useless within D, because of the automaatic
>>preInitialization, which currently eats up about 3 cycles per byte.

Admittedly, I haven't followed processor specs for a while, but initializing memory to zero should take about 1 cycle per 4 bytes, unless I'm totally confused. And on a 64 bit bus machine 1 cycle 8 bytes. (We're talking about long sequences of memory here, not small structs or single variables.)

Does someone know this better?

On 2005-02-01 16:17:17 -0600, Georg Wrede <georg.wrede@nospam.org> said:

>> "Manfred Nowak" <svv1999@hotmail.com> wrote in message
>>> But this technic is useless within D, because of the automaatic
>>> preInitialization, which currently eats up about 3 cycles per byte.
> 
> Admittedly, I haven't followed processor specs for a while, but initializing memory to zero should take about 1 cycle per 4 bytes, unless I'm totally confused. And on a 64 bit bus machine 1 cycle 8 bytes. (We're talking about long sequences of memory here, not small structs or single variables.)

No guys. Cycle counting is as old as MSDOS floppy disks. It's much better and more complicated than that anymore. First of all, memory is transfered from RAM to L1 cache in a cache-line fill. 32 or 64 byte lines depending on the architecture. Then to L2 (on chip) cache. Various RAM types (ie: DDR) and cache read/write strategies will greatly affect how fast this is. Wait states are the problem area. A processor can sit there doing nothing for many "cycles" waiting for the bus and memory controller to get their ass in gear.

Now, Superscalar processors (pentium onward) can read an write in parallel as long as the data does not depend on the previous results. Clearing memory would be such a case. So two ore more write instructions could be paired and executed in one "cycle." And if SIMD instructions are being used, then were talking even more through put. There's just no way of counting anymore. It's an old practice that doesn't relate to current hardware anymore.

For instance, on the PPC it's all about keeping data in the cache. You can have an other wise high "cycle count" of code but if it keeps cache misses to a minimum then it will out preform a piece of code optimized on "cycle counting."

The question is how much throughput and bandwidth do you have? The processor and it's instruction set is not the issue.

But all of this is irrelevant to me, because if you're wanting to do gigabyte bare-metal memory blits for video editing, it would be beyond me why you would be using D and expecting it to do what you want. Why not just use Java? That makes about as much sense. You need to know YOUR hardware and write bare-metal ASM to get what you need done if it's that vital.

I don't use a can opener to peal a grapefruit just because it's brand new and shiny.

> But all of this is irrelevant to me, because if you're wanting to do gigabyte bare-metal memory blits for video editing, it would be beyond me why you would be using D and expecting it to do what you want. Why not just use Java? That makes about as much sense. You need to know YOUR hardware and write bare-metal ASM to get what you need done if it's that vital.

D is aiming to support bare-metal programming, from what I gather from the
Major Goals section of http://www.digitalmars.com/d/overview.html:
"Provide low level bare metal access as required"
D and Java are lightyears apart in terms of bare-metal access.

The existing way to get gobs of uninitialized memory is to call std.c.stdlib.malloc and manage the memory by hand. That's fine and dandy but we want more. What was that old Queen song? "I want it all and I want it now"? Sounds good to me. :-)

One potential solution:  Formalize the existing (or semi-well known) method for
"reserving space":  ie. X.reserve(N) => X.length = N; X.length = old_length;

By creating a real method called "reserve" the array could have its cake and eat it too:  reserve would be required to allocate the memory, but NOT to clear it; that would STILL happen when the length was bumped up, but now it could be done lazy-style.

Additional benefit: objects which override [] could also do reserve(), and could
have special behaviour which is smarter than adjusting length() twice.  In most
cases, they would just pass the savings down by using reserve() instead of
length() on *their* underlying data structure.

I'm including a simple memory bandwidth meter quickie.

Kevin

:
:private import std.date;
:private import std.stdio;
:private import std.conv;
:
:int main(char[][] args)
:{
:    long N = 100;
:    long MB = 1024*1024;
:    long Z1 = 64*MB;
:    long Z = Z1;
:    char[] p;
:
:    if (args.length > 1) {
:        N = toInt(args[1]);
:    }
:
:    if (args.length > 2) {
:        Z = toInt(args[2]) * MB;
:    }
:
:    if (! N) {
:        N = 1;
:    }
:
:    if (Z < 1024) {
:        Z = 256*MB;
:    }
:
:    writef("Looping %s times.\n", N);
:    writef("Writing %s bytes/loop.\n", Z);
:
:    d_time t1 = getUTCtime();
:
:    for(int i = 0; i<N; i++) {
:        p.length = Z;
:        if (p[p.length / 3] == 'c') {
:            writef("Have C\n");
:        }
:
:        if (p[p.length - 1] == 'q') {
:            writef("Have Q\n");
:        }
:
:        p[p.length / 3] = 'c';
:        p[p.length - 1] = 'q';
:
:        p.length = 1234;
:    }
:
:    d_time t2 = getUTCtime();
:
:    double sec = ((t2-t1) + 0.0)/TicksPerSecond;
:
:    writef("Time elapsed = %s [res=%s/s].\n",
:           sec, TicksPerSecond);
:
:    writef("Mem b/w = %s MB / sec.\n", ((Z/MB)*N)/sec);
:
:    return 0;
:}
: