On Thursday, 23 May 2013 at 20:51:42 UTC, Brad Anderson wrote:
> On Thursday, 23 May 2013 at 20:02:06 UTC, QAston wrote:
>> I think that Phobos should have some support for manual memory management. I don't mean clearing out the gc usage there, as it's fairly obvious. I rather think about something like unique_ptr/shared_ptr in the std. I think unique_ptr can't be implemented without rval refs, also C++ sollutions may not fit here. Anyways, now it's not so straightforward how to live without gc so standard sollution would be really helpful.
>
> There is std.typecons.Unique and std.typecons.RefCounted.  Unique is more cumbersome than unique_ptr but it should work though I've never tried to use it.  Proper rvalue references would be a nice improvement here.
>
> RefCounted doesn't support classes yet simply because nobody has taken the time to add support for them.
>
> It'd be nice to just be able to say shared_ptr = RefCounted, unique_ptr = Unique when somebody asks about smart pointers in D though.
>
> std.typecons.scoped is also useful but a bit buggy/cumbersome.  jA_cOp (IRC handle) is working on improving it.  Manu tried his hand at implementing his own version for fun (which came up because we were engaged in yet another GC argument with someone coming from C++).

Thank you very much for the reply - I didn't realize those were already there.

Am 23.05.2013 20:13, schrieb Brad Anderson:
> While there hasn't been anything official, I think it's a safe bet to
> say that D is being used for a major title, Remedy's Quantum Break,
> featured prominently during the announcement of Xbox One. Quantum Break
> doesn't come out until 2014 so the timeline seems about right (Remedy
> doesn't appear to work on more than one game at a time from what I can
> tell).
>
>
> That's pretty huge news.
>
>
> Now I'm wondering what can be done to foster this newly acquired
> credibility in games.  By far the biggest issue I hear about when it
> comes to people working on games in D is the garbage collector.  You can
> work around the GC without too much difficulty as Manu's experience
> shared in his DConf talk shows but a lot of people new to D don't know
> how to do that.  We could also use some tools and guides to help people
> identify and avoid GC use when necessary.
>
> @nogc comes to mind (I believe Andrei mentioned it during one of the
> talks released). [1][2]
>
> Johannes Pfau's work in progress -vgc command line option [3] would be
> another great tool that would help people identify GC allocations.  This
> or something similar could also be used to document throughout phobos
> when GC allocations can happen (and help eliminate it where it makes
> sense to).
>
> There was a lot of interesting stuff in Benjamin Thaut's article about
> GC versus manual memory management in a game [4] and the discussion
> about it on the forums [5].  A lot of this collective knowledge built up
> on manual memory management techniques specific to D should probably be
> formalized and added to the official documentation.  There is a Memory
> Management [6] page in the documentation but it appears to be rather
> dated at this point and not particularly applicable to modern D2 (no
> mention of emplace or scoped and it talks about using delete and scope
> classes).
>
> Game development is one place D can really get a foothold but all too
> often the GC is held over D's head because people taking their first
> look at D don't know how to avoid using it and often don't realize you
> can avoid using it entirely. This is easily the most common issue raised
> by newcomers to D with a C or C++ background that I see in the #d IRC
> channel (many of which are interested in game dev but concerned the GC
> will kill their game's performance).
>
>
> 1: http://d.puremagic.com/issues/show_bug.cgi?id=5219
> 2: http://wiki.dlang.org/DIP18
> 3: https://github.com/D-Programming-Language/dmd/pull/1886
> 4: http://3d.benjamin-thaut.de/?p=20#more-20
> 5: http://forum.dlang.org/post/k27bh7$t7f$1@digitalmars.com
> 6: http://dlang.org/memory.html

With the increase usage of Windows Phone 8 (I know I know), MonoGame, Unity3D and now PS Vita, C# usage is also increasing, so there is a risk of D losing that train, even with Remedy's good example.

Please note that on those scenarios, C# is also compiled to native code.

--
Paulo

On 05/23/2013 08:43 PM, H. S. Teoh wrote:
> I listened to Manu's talk yesterday, and I agree with what he said, that Phobos functions that don't *need* to allocate, shouldn't. Andrei was also enthusiastic about std.algorithm being almost completely allocation-free. Maybe we should file bugs (enhancement requests?) for all such Phobos functions?

I'm also in agreement with Manu.  There may well already be bugs for some of them -- e.g. there is one for toUpperInPlace which he referred to, and the source of the allocation is clear and is even responsible for other bugs: http://d.puremagic.com/issues/show_bug.cgi?id=9629

I asked for a list because, even if all the cases are registered as bugs, it's not necessarily easy to find them.  So, we need to either tag all the bugs so they can be found easily, or make a list somewhere.

On Thursday, May 23, 2013 22:02:05 QAston wrote:
> I think that Phobos should have some support for manual memory management. I don't mean clearing out the gc usage there, as it's fairly obvious. I rather think about something like unique_ptr/shared_ptr in the std. I think unique_ptr can't be implemented without rval refs, also C++ sollutions may not fit here. Anyways, now it's not so straightforward how to live without gc so standard sollution would be really helpful.

We have std.typecons.RefCounted, which is basically a shared pointer.

> Also, it should be visible in C++/D that D can really deal with manual memory management conveniently - when I checked out Dlang first time I felt very disappointed that "delete" operator is deprecated. "So - they advertise one can code without GC, yet they seem to deprecate the operator" - false claims discourage people from using new languages.

delete is only used for GC memory, and manual memory management should really be done with malloc and free rather than explicitly freeing GC memory. But if you really want to risk blowing your foot off, you can always use destroy to destroy an object in GC memory and core.memory.GC.free to free it.

Also, once we get custom allocators, it should be easier to manually manage memory (e.g. I would assume that it would properly abstract doing malloc and then emplacing the object in that memory so that you do something like allocator!MyObject(args) rather than having to deal with emplace directly).

- Jonathan M Davis

On 24 May 2013 05:02, Szymon Gatner <noemail@gmail.com> wrote:

> On Thursday, 23 May 2013 at 18:50:11 UTC, Brad Anderson wrote:
>
>> On Thursday, 23 May 2013 at 18:43:01 UTC, Szymon Gatner wrote:
>>
>>> On Thursday, 23 May 2013 at 18:13:17 UTC, Brad Anderson wrote:
>>>
>>>> While there hasn't been anything official, I think it's a safe bet to say that D is being used for a major title, Remedy's Quantum Break, featured prominently during the announcement of
>>>>
>>>
>>> May I ask where this intel comes from? Do you have any more details on how D is used in the project?
>>>
>>
>> You can watch Manu's talk from DConf here:
>>
>> http://www.youtube.com/watch?**v=FKceA691Wcg<http://www.youtube.com/watch?v=FKceA691Wcg>
>>
>> tl;dw They are using it as a rapid turn around scripting language for their C++ engine.
>>
>
> Ah I did watch it. Didn't realize Manu works at Remedy. Being small indie game dev I totally agree on the industry needing salvation from C++.
>
> I am watching D closely for few years now but until compiler is more stable (tho this is less and less of a problem) and there is decent ARM support I still can't allow myself to switch. And the day of the switch will be glorious one.
>

I really hope D on ARM gets some more attention in the near future. The day it can be used on Android will be a very significant breakthrough!

On 24 May 2013 05:37, Kiith-Sa <kiithsacmp@gmail.com> wrote:

> On Thursday, 23 May 2013 at 18:13:17 UTC, Brad Anderson wrote:
>
>> While there hasn't been anything official, I think it's a safe bet to say that D is being used for a major title, Remedy's Quantum Break, featured prominently during the announcement of Xbox One. Quantum Break doesn't come out until 2014 so the timeline seems about right (Remedy doesn't appear to work on more than one game at a time from what I can tell).
>>
>>
>> That's pretty huge news.
>>
>>
>> Now I'm wondering what can be done to foster this newly acquired credibility in games.  By far the biggest issue I hear about when it comes to people working on games in D is the garbage collector.  You can work around the GC without too much difficulty as Manu's experience shared in his DConf talk shows but a lot of people new to D don't know how to do that.  We could also use some tools and guides to help people identify and avoid GC use when necessary.
>>
>> @nogc comes to mind (I believe Andrei mentioned it during one of the
>> talks released). [1][2]
>>
>> Johannes Pfau's work in progress -vgc command line option [3] would be another great tool that would help people identify GC allocations.  This or something similar could also be used to document throughout phobos when GC allocations can happen (and help eliminate it where it makes sense to).
>>
>> There was a lot of interesting stuff in Benjamin Thaut's article about GC versus manual memory management in a game [4] and the discussion about it on the forums [5].  A lot of this collective knowledge built up on manual memory management techniques specific to D should probably be formalized and added to the official documentation.  There is a Memory Management [6] page in the documentation but it appears to be rather dated at this point and not particularly applicable to modern D2 (no mention of emplace or scoped and it talks about using delete and scope classes).
>>
>> Game development is one place D can really get a foothold but all too often the GC is held over D's head because people taking their first look at D don't know how to avoid using it and often don't realize you can avoid using it entirely. This is easily the most common issue raised by newcomers to D with a C or C++ background that I see in the #d IRC channel (many of which are interested in game dev but concerned the GC will kill their game's performance).
>>
>>
>> 1: http://d.puremagic.com/issues/**show_bug.cgi?id=5219<http://d.puremagic.com/issues/show_bug.cgi?id=5219>
>> 2: http://wiki.dlang.org/DIP18
>> 3: https://github.com/D-**Programming-Language/dmd/pull/**1886<https://github.com/D-Programming-Language/dmd/pull/1886>
>> 4: http://3d.benjamin-thaut.de/?**p=20#more-20<http://3d.benjamin-thaut.de/?p=20#more-20>
>> 5: http://forum.dlang.org/post/**k27bh7$t7f$1@digitalmars.com<http://forum.dlang.org/post/k27bh7$t7f$1@digitalmars.com>
>> 6: http://dlang.org/memory.html
>>
>
>
> Without official confirmation, I think it's rather early to assume D's being used in Quantum Break. D might compile on the new consoles, but what about druntime/phobos/etc ?
>
> That said, I support this idea.
>
> When I get time I'll try looking at Phobos if there is some low-hanging fruit with regards to GC usage and submit pull requests (I didn't make any non-doc contribution to Phobos yet, but I have a general idea of how its source looks).
>
> I also think that many people overreact about GC too much. @nogc is
> certainly a good idea, but I think strategically using malloc,
> disabling/reenabling GC, using GC.free and even just using standard GC
> features *while taking care to
> avoid unnecessary allocations* is vastly better than outright removing GC.
>

Just to be clear, while I've hard many have, I've NEVER argued for removing
the GC. I think that's a hallmark of a modern language. I want to use the
GC in games, but it needs to have performance characteristics that are
applicable to realtime and embedded use.
Those are:
1. Can't stop the world.
2. Needs tight controls, enable/disable, and the allocators interface so
alternative memory sources can be used in mane places.
3. Needs to (somehow) run incrementally. I'm happy to budget a few hundred
µs per frame, but not a millisecond every 10 frames, or 1 second every 1000.
    It can have 1-2% of overall frame time each frame, but it can't have
10-100% of random frames here and there. This results in framerate spikes.

The GC its self can be much less efficient than the existing GC if it
want's, it's only important that it can be halted at fine grained
intervals, and that it will eventually complete its collect cycle over the
long-term.
I know that an incremental GC like this is very complex, but I've never
heard of any real experiments, so maybe it's not impossible?

It'd be good to have an easy-to-use way to manually allocate
> classes/structs in Phobos (higher-level than emplace, something close in usability to C++ new/delete), preferably with a way to override the allocation mechanism (I assume the fabled "allocators" have something to do with this? Maybe we'll get them once DNF is released... ... ...)
>

On 24 May 2013 09:02, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Thursday, May 23, 2013 22:02:05 QAston wrote:
> > I think that Phobos should have some support for manual memory management. I don't mean clearing out the gc usage there, as it's fairly obvious. I rather think about something like unique_ptr/shared_ptr in the std. I think unique_ptr can't be implemented without rval refs, also C++ sollutions may not fit here. Anyways, now it's not so straightforward how to live without gc so standard sollution would be really helpful.
>
> We have std.typecons.RefCounted, which is basically a shared pointer.
>

I've always steered away from things like this because it creates a
double-indirection.
I have thought of making a similar RefCounted template, but where the
refCount is stored in a hash table, and the pointer is used to index the
table.
This means the refCount doesn't pollute the class/structure being
ref-counted, or avoids a double-indirection on general access.
It will be slightly slower to inc/decrement, but that's a controlled
operation.
I would use a system like this for probably 80% of resources.

> Also, it should be visible in C++/D that D can really deal with
> > manual memory management conveniently - when I checked out Dlang first time I felt very disappointed that "delete" operator is deprecated. "So - they advertise one can code without GC, yet they seem to deprecate the operator" - false claims discourage people from using new languages.
>
> delete is only used for GC memory, and manual memory management should
> really
> be done with malloc and free rather than explicitly freeing GC memory. But
> if
> you really want to risk blowing your foot off, you can always use destroy
> to
> destroy an object in GC memory and core.memory.GC.free to free it.
>
> Also, once we get custom allocators, it should be easier to manually manage
> memory (e.g. I would assume that it would properly abstract doing malloc
> and
> then emplacing the object in that memory so that you do something like
> allocator!MyObject(args) rather than having to deal with emplace directly).
>

Custom allocators will probably be very useful, but if there's one thing
STL has taught me, it's hard to use them effectively, and in practise,
nobody ever uses them.
One problem is the implicit allocation functions (array concatenation,
AA's, etc). How to force those to allocate somewhere else for the scope?


> - Jonathan M Davis
>

On 05/24/2013 01:25 AM, Manu wrote:
> I really hope D on ARM gets some more attention in the near future. The day it can be used on Android will be a very significant breakthrough!

GDC is close to being fully usable on ARM, no?  And as I recall the only (albeit major) problem you had with GDC was the delay between bugfixes landing in the D frontend and carrying over to GDC.

So, the solution here might be the work to properly generalize the frontend so that it will plug-and-play on top of any of the available backends.

On 05/24/2013 01:34 AM, Manu wrote:
> Just to be clear, while I've hard many have, I've NEVER argued for removing the
> GC. I think that's a hallmark of a modern language. I want to use the GC in
> games, but it needs to have performance characteristics that are applicable to
> realtime and embedded use.
> Those are:
> 1. Can't stop the world.
> 2. Needs tight controls, enable/disable, and the allocators interface so
> alternative memory sources can be used in mane places.
> 3. Needs to (somehow) run incrementally. I'm happy to budget a few hundred µs
> per frame, but not a millisecond every 10 frames, or 1 second every 1000.
>     It can have 1-2% of overall frame time each frame, but it can't have 10-100%
> of random frames here and there. This results in framerate spikes.
> 
> The GC its self can be much less efficient than the existing GC if it want's,
> it's only important that it can be halted at fine grained intervals, and that it
> will eventually complete its collect cycle over the long-term.
> I know that an incremental GC like this is very complex, but I've never heard of
> any real experiments, so maybe it's not impossible?

Maybe someone else can point to an example, but I can't think of any language prior to D that has both the precision and speed to be useful for games and embedded programming, and that also has GC built in.

So it seems to me that this might well be an entirely new problem, as no other GC language or library has had the motivation to create something that satisfies these use parameters.

This also seems to suggest that an ideal solution might be to have several different GC strategies, the choice of which could be made at compile time depending on what's most suitable for the application in question.

On Friday, May 24, 2013 09:42:10 Manu wrote:
> On 24 May 2013 09:02, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > On Thursday, May 23, 2013 22:02:05 QAston wrote:
> > > I think that Phobos should have some support for manual memory management. I don't mean clearing out the gc usage there, as it's fairly obvious. I rather think about something like unique_ptr/shared_ptr in the std. I think unique_ptr can't be implemented without rval refs, also C++ sollutions may not fit here. Anyways, now it's not so straightforward how to live without gc so standard sollution would be really helpful.
> > 
> > We have std.typecons.RefCounted, which is basically a shared pointer.
> 
> I've always steered away from things like this because it creates a
> double-indirection.
> I have thought of making a similar RefCounted template, but where the
> refCount is stored in a hash table, and the pointer is used to index the
> table.
> This means the refCount doesn't pollute the class/structure being
> ref-counted, or avoids a double-indirection on general access.
> It will be slightly slower to inc/decrement, but that's a controlled
> operation.
> I would use a system like this for probably 80% of resources.

We use smart pointers where I work and it's a godsend for avoiding memory problems. We almost never have them whereas the idiots who designed the older software used manual refcounting everywhere, and they had tons of memory problems. But while we need to be performant, we don't need to be performant on quite the level that you do. So, maybe it's more of a problem in your environment.

> > Also, it should be visible in C++/D that D can really deal with
> > 
> > > manual memory management conveniently - when I checked out Dlang first time I felt very disappointed that "delete" operator is deprecated. "So - they advertise one can code without GC, yet they seem to deprecate the operator" - false claims discourage people from using new languages.
> > 
> > delete is only used for GC memory, and manual memory management should
> > really
> > be done with malloc and free rather than explicitly freeing GC memory. But
> > if
> > you really want to risk blowing your foot off, you can always use destroy
> > to
> > destroy an object in GC memory and core.memory.GC.free to free it.
> > 
> > Also, once we get custom allocators, it should be easier to manually
> > manage
> > memory (e.g. I would assume that it would properly abstract doing malloc
> > and
> > then emplacing the object in that memory so that you do something like
> > allocator!MyObject(args) rather than having to deal with emplace
> > directly).
> 
> Custom allocators will probably be very useful, but if there's one thing STL has taught me, it's hard to use them effectively, and in practise, nobody ever uses them.

Well, as Andrei said, they're hard, which is why they aren't done yet. Another think to think about with regards to C++ though is the fact that its new and delete don't having anything to do with a GC, so it has a built-in nice way of allocating memory which is managed manually, whereas in D, we're forced to use emplace, which is a lot more of a hassle. Even simply having something like allocator.make!MyObj(args) and allocator.free(args) would really help out. There's no question though that they get hairier when you start having to worry about containers and internal allocations and the like. It's a tough problem.

> One problem is the implicit allocation functions (array concatenation, AA's, etc). How to force those to allocate somewhere else for the scope?

I would fully expect that they use the GC and only the GC as they're language constructs, and custom allocators are going to be library constructs. The allocators may provide clean ways to do stuff like concatenating arrays using their API rather than ~, but if you really want to manipulate arrays with slicing and concatenation and whatnot without the GC, I think that you're pretty much going to have to create a new type to handle them, which is very doable, but it does mean not using the built-in arrays as much, which does kind of suck. But for most programs, I would expect that simply managing the GC more intelligently for stuff that has to be GC allocated would solve the problem nicely. Kiith-Sa and others have managed to quite well at getting the GC to work efficiently by managing when it's enabled and gets the chance to run and whatnot. You have extremely stringent requirements that may cause problems with that (though Kiith-Sa was doing a game of some variety IIRC), but pretty much the only way to make it so that built-in stuff that allocates doesn't use the GC is to use your own version of druntime.

Kiith-Sa had a good post on how to go about dealing with the GC in performant code a while back:

http://forum.dlang.org/post/vbsajlgotanuhmmpnspf@forum.dlang.org

Regardless, we're not going to get away from some language features requiring the GC, but they're also not features that exist in C++, so if you really can't use them, you still haven't lost anything over C++ (as much as it may still suck to not be able to them), and there are still plenty of other great features that you can take advantage of.

- Jonathan M Davis