On 1/7/2012 5:54 PM, Manu wrote:
> On 8 January 2012 03:46, Walter Bright <newshound2@digitalmars.com
> <mailto:newshound2@digitalmars.com>> wrote:
>
>     It should also be possible to do the gc in another thread running at low
>     priority, if the only other running thread is doing the critical stuff and
>     is not the sole holder of references to any GC allocated memory it is using.
>
>
> ... sound like unreliable restrictions. Now sure I follow exactly what you mean
> by 'only other running thread', and 'critical stuff'.

The gc pauses all threads to run the collection cycle. But if a thread does not uniquely hold roots to gc memory, then it can be left running. The gc thread can then be made low priority, and the other running thread(s) high priority. Presumably those other running threads would then be the time critical ones.

On 1/7/2012 4:12 PM, Peter Alexander wrote:
> The main advantage of Lua for game code (in my opinion) is runtime reloading,
> and the ability to avoid recompiles just to test some new game logic. That's not
> so easy with C++.

D is far faster at compiling than C++.

On 01/07/2012 08:47 PM, Manu wrote:
> On 8 January 2012 03:40, Nick Sabalausky <a@a.a> wrote:
> 
>     "Froglegs" <lugtug@gmail.com <mailto:lugtug@gmail.com>> wrote in message
>     news:lwcqnrvamqlnjjlxzbqa@dfeed.kimsufi.thecybershadow.net...
>     >
>     > can reload scripts while program is executing. Neither D nor C++ work
>     > here.
>     >
> 
>     Why not a dll? Those can be compiled/loaded/reloaded at runtime. And
>     since
>     it's just scripts, it wouldn't take long to compile at all, at least
>     in D
>     (maybe not so much in C++).
> 
> 
> It's common for designers to edit scripts, and they don't usually have coding environments on their machines.

Meh, just link/package a D compiler into/with the game and recompile/reload a file whenever it changes.  It would probably help if DMD ditched its non-redistributable license stuff, but there's also LDC/GDC (LDC is probably the most promising for this kind of thing).

Chad J wrote:
> On 01/07/2012 08:47 PM, Manu wrote:
>> On 8 January 2012 03:40, Nick Sabalausky <a@a.a> wrote:
>> 
>>  "Froglegs" <lugtug@gmail.com <mailto:lugtug@gmail.com>> wrote in message
>>  news:lwcqnrvamqlnjjlxzbqa@dfeed.kimsufi.thecybershadow.net...
>>  >
>>  > can reload scripts while program is executing. Neither D nor C++ work
>>  > here.
>>  >
>> 
>>  Why not a dll? Those can be compiled/loaded/reloaded at runtime. And
>>  since
>>  it's just scripts, it wouldn't take long to compile at all, at least
>>  in D
>>  (maybe not so much in C++).
>> 
>> 
>> It's common for designers to edit scripts, and they don't usually have
>> coding environments on their machines.
>
> Meh, just link/package a D compiler into/with the game and
> recompile/reload a file whenever it changes.  It would probably help if
> DMD ditched its non-redistributable license stuff, but there's also
> LDC/GDC (LDC is probably the most promising for this kind of thing).

My thoughts exactly. A D compiler/linker should be able to be packaged with an game engine editor rather easily, and if I'm not mistaken LDC has the potential to offer JIT/AOT compiling.




Manu wrote:
> I'm generally shovelling large, strictly controlled buffers which need
> proper placement/alignment, pool allocation, or allocation from temporal
> ring buffers and that sort of thing. I don't typically find C/C++
> intolerable, or even particularly unpleasant for engine level code as it
> is, it's the higher level code I'm keen to escape.

As I understand it, most low level game engine code deals with reusable memory pools to reduce memory thrashing with large amounts of dynamically allocated/deallocated objects (correct me if I'm wrong). C/C++ is great for this because in C memory is memory, whereas in C#/Java memory is objects. Which makes general purpose memory pools difficult to manage. I know there are engines like the Delta Engine which, to my knowledge, is written entirely in C# (minus a few minor bits).
I'm currently porting over our Reign SDK (code.google.com/p/reign-sdk) which is a modest set of libraries attempting to wrap DirectX, OpenGL, and other 3D media libraries into a universal interface. I've plans on enhancing, d-atizing, and expanding these libraries into a more full featured game engine one day. I've got some interesting ideas on how pre-written code packages could be easily designer-style assembled in-editor and compiled into efficient native logic blocks "on the fly". Only D's fast native compile times and easy-to-grasp syntax would really allow for what I'm thinking.

Given your experience in this area, I would appreciate any insight you could offer about the potential pros/cons for writing low level game engine components in D. Would you say D is a effective tool to write a general purpose memory pool, or would something like that be better written in C? Or.. is it common to have an array of specialized object pools? I'd imagine such an engine would sacrifice flexibility and eat up more memory, but most likely easier to implement.

On 08-01-2012 00:06, Manu wrote:
> On 7 January 2012 20:59, bearophile <bearophileHUGS@lycos.com
> <mailto:bearophileHUGS@lycos.com>> wrote:
>
>     Manu:
>
>      > The tendency to encourage use of dynamic arrays will be a major
>     problem.
>
>     I don't know how much big that problem will be, D dynamic arrays are
>     quite handy, but I agree that static arrays need to be encouraged
>     more in D (and currently most algorithms of Phobos don't work with
>     static arrays (you need to slice them first)).
>
>
> A slice doesn't produce a GC allocation does it?
> I thought a slice was just a pointer-length pair. Should live on the
> stack/in regs?

AFAIK yes.

>
> ...so slicing static arrays shouldn't be a problem right?
>
>     Currently even this code with a stack-allocated fixed size array
>     causes a heap allocation (DMD):
>
>     void main() {
>         int[2] a = [1, 2];
>     }
>
>     ASM, optimized build:
>
>     __Dmain comdat
>         push EAX
>         push EAX
>         mov EAX,offset FLAT:_D12TypeInfo_xAi6__initZ
>         push EBX
>         push 8
>         push 2
>         push EAX
>         call near ptr __d_arrayliteralTX ; heap allocation
>         add ESP,8
>         mov EBX,EAX
>         mov dword ptr [EAX],1
>         mov ECX,EBX
>         push EBX
>         lea EDX,0Ch[ESP]
>         mov dword ptr 4[EBX],2
>         push EDX
>         call near ptr _memcpy
>         add ESP,0Ch
>         xor EAX,EAX
>         pop EBX
>         add ESP,8
>         ret
>
>
> What the hell is it allocating?
> Surely that's not necessary... that's gotta be fixable?
>
>     I have suggested to add a safer version of VLAs to avoid some
>     heap-allocated dynamic arrays:
>     http://d.puremagic.com/issues/show_bug.cgi?id=5348
>
>
> +1!!
> I'm surprised this already isn't supported!


-- 
- Alex

On 8 January 2012 08:03, F i L <witte2008@gmail.com> wrote:

> I've got some interesting ideas on how pre-written code packages could be easily designer-style assembled in-editor and compiled into efficient native logic blocks "on the fly". Only D's fast native compile times and easy-to-grasp syntax would really allow for what I'm thinking.
>

If you're talking about stringing together pre-written code blocks with
some editor, then this might be an interesting approach.
I'm not so sure how 'easy-to-grasp' D is, or why that's important if you're
providing an editor?
I tend to think D is considerably less simple than C, possibly more complex
(but less archaic) than C++... It's not really something you could expose
to a designer.


> Given your experience in this area, I would appreciate any insight you could offer about the potential pros/cons for writing low level game engine components in D. Would you say D is a effective tool to write a general purpose memory pool, or would something like that be better written in C?


I'm probably not the best to comment, because I still haven't internalised
all the possibilities of D's powerful language features.
But I think C is basically a subset of D, so I don't see any inhibiting
reason why it couldn't all be done in D. My concern is mainly over whether
D makes it easy, or ugly to manage all the resources as strictly as
required, and how much of D's idioms/paradigms you need to subvert to
actually do it.

Writing an engine in 'C-ish' D is surely possible, writing it in 'D'...
potentially problematic.
I'm keen to have a go in the near future as an experiment, I expect in
engine level code, the GC might frustrate me.
Writing various memory managers/pools/buffers themselves is surely fine in
D, no problem... but USING them in an idiomatic way to allocate objects,
there's no support in D.

D:
MyObject obj = new MyObject(x, y, z);

D (subverting GC):
MyObject* obj = (MyObject*)someManager.Alloc(MyObject.sizeof);
....? how do I even perform a placement new?

I wouldn't want to be doing that everywhere.
If D implements allocators, then I can see that being much better, enabling
one to still write fairly conventional D code.

Or.. is it common to have an array of specialized object pools? I'd imagine
> such an engine would sacrifice flexibility and eat up more memory, but most likely easier to implement.


I fail to see where this is sacrificing flexibility, but yes, you do tend to over-reserve in these sorts of pools, but there's no real alternative (although I use a lot of tricks).

Cache efficiently is all about memory locality, and pooling 'like' things
together and stream processing them in batches is the most efficient way to
do work on ANY computer.
Once you write your systems this way, they tend to thread+scale well, and
automatically. You could even easily offload them to a video card, or other
sort of DSP.

What I usually do to avoid major over-reservation of memory is to break the
pools into smaller 'buckets' (some multiple-of and aligned-to the systems
most course grained cache), and extend/contract the pool size as needed.
Cache alignment is very important, I was surprised+concerned the other
night when Walter mentioned that D does NOT support aligning anything to
any power of 2 using the align() attribute.
I'm curious to know under what circumstances it actually works?

On 8/01/12 4:11 AM, Walter Bright wrote:
> On 1/7/2012 4:12 PM, Peter Alexander wrote:
>> The main advantage of Lua for game code (in my opinion) is runtime
>> reloading,
>> and the ability to avoid recompiles just to test some new game logic.
>> That's not
>> so easy with C++.
>
> D is far faster at compiling than C++.

True, but:

- It's not orders of magnitude faster. You still have to wait for a re-compile of large projects.

- Linking takes just as much time as compiling.

- Can't account for the time it takes to boot your application back up after compiling and get back into the state you were in when you wanted to make the change.

- Also, DMD is slow at optimising. http://d.puremagic.com/issues/show_bug.cgi?id=7157

On 2012-01-08 02:40, Nick Sabalausky wrote:
> "Froglegs"<lugtug@gmail.com>  wrote in message
> news:lwcqnrvamqlnjjlxzbqa@dfeed.kimsufi.thecybershadow.net...
>>
>> can reload scripts while program is executing. Neither D nor C++ work
>> here.
>>
>
> Why not a dll? Those can be compiled/loaded/reloaded at runtime. And since
> it's just scripts, it wouldn't take long to compile at all, at least in D
> (maybe not so much in C++).

Yeah, that's pretty cool. This was show at the Tango conference: http://vimeo.com/2264486 around 30:35

-- 
/Jacob Carlborg

Am 08.01.2012 12:05, schrieb Manu:
> On 8 January 2012 08:03, F i L <witte2008@gmail.com
> <mailto:witte2008@gmail.com>> wrote:
>
>     I've got some interesting ideas on how pre-written code packages
>     could be easily designer-style assembled in-editor and compiled into
>     efficient native logic blocks "on the fly". Only D's fast native
>     compile times and easy-to-grasp syntax would really allow for what
>     I'm thinking.
>
>
> If you're talking about stringing together pre-written code blocks with
> some editor, then this might be an interesting approach.
> I'm not so sure how 'easy-to-grasp' D is, or why that's important if
> you're providing an editor?
> I tend to think D is considerably less simple than C, possibly more
> complex (but less archaic) than C++... It's not really something you
> could expose to a designer.
>
>     Given your experience in this area, I would appreciate any insight
>     you could offer about the potential pros/cons for writing low level
>     game engine components in D. Would you say D is a effective tool to
>     write a general purpose memory pool, or would something like that be
>     better written in C?
>
>
> I'm probably not the best to comment, because I still haven't
> internalised all the possibilities of D's powerful language features.
> But I think C is basically a subset of D, so I don't see any inhibiting
> reason why it couldn't all be done in D. My concern is mainly over
> whether D makes it easy, or ugly to manage all the resources as strictly
> as required, and how much of D's idioms/paradigms you need to subvert to
> actually do it.
>
> Writing an engine in 'C-ish' D is surely possible, writing it in 'D'...
> potentially problematic.
> I'm keen to have a go in the near future as an experiment, I expect in
> engine level code, the GC might frustrate me.
> Writing various memory managers/pools/buffers themselves is surely fine
> in D, no problem... but USING them in an idiomatic way to allocate
> objects, there's no support in D.
>
> D:
> MyObject obj = new MyObject(x, y, z);
>
> D (subverting GC):
> MyObject* obj = (MyObject*)someManager.Alloc(MyObject.sizeof);
> ....? how do I even perform a placement new?
>

Just an idea how it could look whith allocators: What about adding an argument to the new operator for the allocator struct (like GC), and let the user define an default one ? I think something similar can already be done by overloading the new  and assignment operator:

Runtime.setDefaultAllocator!GC();
auto refcounted = new!RefCounter MyObject();

> I wouldn't want to be doing that everywhere.
> If D implements allocators, then I can see that being much better,
> enabling one to still write fairly conventional D code.
>
>     Or.. is it common to have an array of specialized object pools? I'd
>     imagine such an engine would sacrifice flexibility and eat up more
>     memory, but most likely easier to implement.
>
>
> I fail to see where this is sacrificing flexibility, but yes, you do
> tend to over-reserve in these sorts of pools, but there's no real
> alternative (although I use a lot of tricks).
>
> Cache efficiently is all about memory locality, and pooling 'like'
> things together and stream processing them in batches is the most
> efficient way to do work on ANY computer.
> Once you write your systems this way, they tend to thread+scale well,
> and automatically.. You could even easily offload them to a video card,
> or other sort of DSP.
>
> What I usually do to avoid major over-reservation of memory is to break
> the pools into smaller 'buckets' (some multiple-of and aligned-to the
> systems most course grained cache), and extend/contract the pool size as
> needed.
> Cache alignment is very important, I was surprised+concerned the other
> night when Walter mentioned that D does NOT support aligning anything to
> any power of 2 using the align() attribute.
> I'm curious to know under what circumstances it actually works?

On 1/8/2012 4:01 AM, Peter Alexander wrote:
> On 8/01/12 4:11 AM, Walter Bright wrote:
>> On 1/7/2012 4:12 PM, Peter Alexander wrote:
>>> The main advantage of Lua for game code (in my opinion) is runtime
>>> reloading,
>>> and the ability to avoid recompiles just to test some new game logic.
>>> That's not
>>> so easy with C++.
>>
>> D is far faster at compiling than C++.
>
> True, but:
>
> - It's not orders of magnitude faster. You still have to wait for a re-compile
> of large projects.
>
> - Linking takes just as much time as compiling.
>
> - Can't account for the time it takes to boot your application back up after
> compiling and get back into the state you were in when you wanted to make the
> change.

All true.

> - Also, DMD is slow at optimising.
> http://d.puremagic.com/issues/show_bug.cgi?id=7157

When you're in the edit-compile-debug loop, it's normal to turn off the optimizer.