On 24 May 2013 09:44, Joseph Rushton Wakeling <joseph.wakeling@webdrake.net>wrote:

> On 05/24/2013 01:25 AM, Manu wrote:
> > I really hope D on ARM gets some more attention in the near future. The
> day it
> > can be used on Android will be a very significant breakthrough!
>
> GDC is close to being fully usable on ARM, no?  And as I recall the only
> (albeit
> major) problem you had with GDC was the delay between bugfixes landing in
> the D
> frontend and carrying over to GDC.
>
> So, the solution here might be the work to properly generalize the
> frontend so
> that it will plug-and-play on top of any of the available backends.
>

Well the compiler seems fine actually. It generates good ARM code in my experience, ditto for PPC, MIPS, SH4 (those are all I have tested).

Druntime needs to be ported to Bionic. People have made a start, but I
recall mention of some complications that need some work?
iOS needs extern(ObjC), but it's a fairly standard posix underneath, so
should be less work on the runtime.

Systems like WiiU/Wii/PS3/XBox360, etc all need runtimes, and those will
probably not be developed by the D community.
It would land on a general gamedev's shoulders to do those, so I would
suggest the approach here would be to make a step-buy-step guide to porting
druntime. Make the process as simple as possible for individuals wanting to
support other 'niche' platforms...

On 24 May 2013 09:57, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Friday, May 24, 2013 09:42:10 Manu wrote:
> > On 24 May 2013 09:02, Jonathan M Davis <jmdavisProg@gmx.com> wrote:
> > > On Thursday, May 23, 2013 22:02:05 QAston wrote:
> > > > I think that Phobos should have some support for manual memory management. I don't mean clearing out the gc usage there, as it's fairly obvious. I rather think about something like unique_ptr/shared_ptr in the std. I think unique_ptr can't be implemented without rval refs, also C++ sollutions may not fit here. Anyways, now it's not so straightforward how to live without gc so standard sollution would be really helpful.
> > >
> > > We have std.typecons.RefCounted, which is basically a shared pointer.
> >
> > I've always steered away from things like this because it creates a
> > double-indirection.
> > I have thought of making a similar RefCounted template, but where the
> > refCount is stored in a hash table, and the pointer is used to index the
> > table.
> > This means the refCount doesn't pollute the class/structure being
> > ref-counted, or avoids a double-indirection on general access.
> > It will be slightly slower to inc/decrement, but that's a controlled
> > operation.
> > I would use a system like this for probably 80% of resources.
>
> We use smart pointers where I work and it's a godsend for avoiding memory
> problems. We almost never have them whereas the idiots who designed the
> older
> software used manual refcounting everywhere, and they had tons of memory
> problems. But while we need to be performant, we don't need to be
> performant
> on quite the level that you do. So, maybe it's more of a problem in your
> environment.
>
> > > Also, it should be visible in C++/D that D can really deal with
> > >
> > > > manual memory management conveniently - when I checked out Dlang first time I felt very disappointed that "delete" operator is deprecated. "So - they advertise one can code without GC, yet they seem to deprecate the operator" - false claims discourage people from using new languages.
> > >
> > > delete is only used for GC memory, and manual memory management should
> > > really
> > > be done with malloc and free rather than explicitly freeing GC memory.
> But
> > > if
> > > you really want to risk blowing your foot off, you can always use
> destroy
> > > to
> > > destroy an object in GC memory and core.memory.GC.free to free it.
> > >
> > > Also, once we get custom allocators, it should be easier to manually
> > > manage
> > > memory (e.g. I would assume that it would properly abstract doing
> malloc
> > > and
> > > then emplacing the object in that memory so that you do something like
> > > allocator!MyObject(args) rather than having to deal with emplace
> > > directly).
> >
> > Custom allocators will probably be very useful, but if there's one thing STL has taught me, it's hard to use them effectively, and in practise, nobody ever uses them.
>
> Well, as Andrei said, they're hard, which is why they aren't done yet.
> Another
> think to think about with regards to C++ though is the fact that its new
> and
> delete don't having anything to do with a GC, so it has a built-in nice
> way of
> allocating memory which is managed manually, whereas in D, we're forced to
> use
> emplace, which is a lot more of a hassle. Even simply having something like
> allocator.make!MyObj(args) and allocator.free(args) would really help out.
> There's no question though that they get hairier when you start having to
> worry about containers and internal allocations and the like. It's a tough
> problem.
>
> > One problem is the implicit allocation functions (array concatenation, AA's, etc). How to force those to allocate somewhere else for the scope?
>
> I would fully expect that they use the GC and only the GC as they're
> language
> constructs, and custom allocators are going to be library constructs. The
> allocators may provide clean ways to do stuff like concatenating arrays
> using
> their API rather than ~, but if you really want to manipulate arrays with
> slicing and concatenation and whatnot without the GC, I think that you're
> pretty much going to have to create a new type to handle them, which is
> very
> doable, but it does mean not using the built-in arrays as much, which does
> kind of suck. But for most programs, I would expect that simply managing
> the
> GC more intelligently for stuff that has to be GC allocated would solve the
> problem nicely. Kiith-Sa and others have managed to quite well at getting
> the
> GC to work efficiently by managing when it's enabled and gets the chance
> to run
> and whatnot. You have extremely stringent requirements that may cause
> problems
> with that (though Kiith-Sa was doing a game of some variety IIRC), but
> pretty
> much the only way to make it so that built-in stuff that allocates doesn't
> use
> the GC is to use your own version of druntime.
>
> Kiith-Sa had a good post on how to go about dealing with the GC in
> performant
> code a while back:
>
> http://forum.dlang.org/post/vbsajlgotanuhmmpnspf@forum.dlang.org
>
> Regardless, we're not going to get away from some language features
> requiring
> the GC, but they're also not features that exist in C++, so if you really
> can't use them, you still haven't lost anything over C++ (as much as it may
> still suck to not be able to them), and there are still plenty of other
> great
> features that you can take advantage of.
>

/agree, except the issue I raised, when ~ is used in phobos.
That means that function is now off-limits. And there's no way to know
which functions they are...

- Jonathan M Davis
>

On Friday, May 24, 2013 10:11:17 Manu wrote:
> /agree, except the issue I raised, when ~ is used in phobos.
> That means that function is now off-limits. And there's no way to know
> which functions they are...

Yes, we need to look at that. I actually don't think that ~ gets used much (primarily because so much of it uses ranges which don't have ~), but it's something that we need to look out for and address. The suggestion of an @nogc attribute which at least guarantees that new isn't used would be nice for that, since it could potentially both guarantee it and document it. But that woludn't work with templated functions for the most part, since types being used with them might allocate, though we could presumably add attribute inferrence for that so that the functions which call them can be marked with @nogc and be able to know that the functions that they're calling obey that.

My guess is that the functions which are most likely to allocate are those which specifically take strings or arrays, as they _can_ use ~, so they probably need to be examined first, but in some cases, they're also the type of function which may _have_ to allocate, depending on what they're doing. Probably the right approach for that is to track down all of those that are allocting, make it so that any of those that can avoid the allocation do, and then create overloads which take an output range or somesuch for those that have to allocate so that preallocated memory and the like can be used for them. And if we actually have any which can't possibly do anything but allocate, they should be clearly documented as such.

All around though, figuring out how to minimize GC usage in Phobos and enforce that is an open problem which is still very much up for discussion on how best to address (particularly when it's quite easy to introduce inadventant allocations with some stuff). But with everything else that we've had to worry about, optimizations like that haven't been as high a priority as they're going to need to be long term. At some point, we're probably going to need to benchmark stuff more agressively and optimize Phobos in general more, because it's the standard library. And eliminating unnecessary memory allocations definitely goes along with that.

- Jonathan M Davis

On 5/23/13 7:42 PM, Manu wrote:
> I've always steered away from things like this because it creates a
> double-indirection.

There's no double indirection for the payload.

> I have thought of making a similar RefCounted template, but where the
> refCount is stored in a hash table, and the pointer is used to index the
> table.
> This means the refCount doesn't pollute the class/structure being
> ref-counted, or avoids a double-indirection on general access.

But that's worse than non-intrusive refcounting, and way worse than intrusive refcounting (which should be the elective method for classes).

> Custom allocators will probably be very useful, but if there's one thing
> STL has taught me, it's hard to use them effectively, and in practise,
> nobody ever uses them.

Agreed.

> One problem is the implicit allocation functions (array concatenation,
> AA's, etc). How to force those to allocate somewhere else for the scope?

I have some ideas.


Andrei

On 2013-05-23 23:42:10 +0000, Manu <turkeyman@gmail.com> said:

> I have thought of making a similar RefCounted template, but where the
> refCount is stored in a hash table, and the pointer is used to index the
> table.
> This means the refCount doesn't pollute the class/structure being
> ref-counted, or avoids a double-indirection on general access.
> It will be slightly slower to inc/decrement, but that's a controlled
> operation.
> I would use a system like this for probably 80% of resources.

I just want to note that this is exactly how reference counts are handled in Apple's Objective-C implementation, with a spin-lock protecting the table.

Actually, on OS X (but not on iOS) there's 4 tables (if I remember well) and which table to use is determined by bits 4 & 5 of the pointer. It probably helps when you have more cores.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca/

On Friday, 24 May 2013 at 00:06:05 UTC, Manu wrote:
> Systems like WiiU/Wii/PS3/XBox360, etc all need runtimes, and those will
> probably not be developed by the D community.
> It would land on a general gamedev's shoulders to do those, so I would
> suggest the approach here would be to make a step-buy-step guide to porting
> druntime. Make the process as simple as possible for individuals wanting to
> support other 'niche' platforms...

Do you think we could expect those ports to be given back to the community when they get written? Or is it more likely that game studios will keep their ports to themselves?

On 24 May 2013 10:59, Joseph Rushton Wakeling <joseph.wakeling@webdrake.net>wrote:

> On Friday, 24 May 2013 at 00:06:05 UTC, Manu wrote:
>
>> Systems like WiiU/Wii/PS3/XBox360, etc all need runtimes, and those will
>> probably not be developed by the D community.
>> It would land on a general gamedev's shoulders to do those, so I would
>> suggest the approach here would be to make a step-buy-step guide to
>> porting
>> druntime. Make the process as simple as possible for individuals wanting
>> to
>> support other 'niche' platforms...
>>
>
> Do you think we could expect those ports to be given back to the community when they get written? Or is it more likely that game studios will keep their ports to themselves?
>

I'd like to think they'd be made available. I certainly would. But you can never predict what the suits up top will tell you that you have to do.

On Fri, 24 May 2013 02:59:44 +0200
"Joseph Rushton Wakeling" <joseph.wakeling@webdrake.net> wrote:

> On Friday, 24 May 2013 at 00:06:05 UTC, Manu wrote:
> > Systems like WiiU/Wii/PS3/XBox360, etc all need runtimes, and
> > those will
> > probably not be developed by the D community.
> > It would land on a general gamedev's shoulders to do those, so
> > I would
> > suggest the approach here would be to make a step-buy-step
> > guide to porting
> > druntime. Make the process as simple as possible for
> > individuals wanting to
> > support other 'niche' platforms...
> 
> Do you think we could expect those ports to be given back to the community when they get written? Or is it more likely that game studios will keep their ports to themselves?

It would be prohibited by console manufacturer's NDAs/developer-licenses.

If you're an official licensed console developer, you can't provide any console-specific code or technical specs to anyone who isn't also covered by the same licensed developer agreement. I'm sure it could be released to, or shared with, other licensed developers (might be paperwork involved, I dunno), but not to the general community.

Licensed console dev is fairly cloak-and-dagger (minus the dagger, perhaps). Game console manufacturers keep a tight enough grip on their systems to make even Apple blush.

For such a thing to be released back to the "community" it would have to come from the homebrew scene (which AIUI, could then be used by licensed devs too...or at least that was my understanding with GBA, so my info may be out-of-date). An officially licensed developer would lose their license, or get sued, or something.

Am Thu, 23 May 2013 20:21:47 -0400
schrieb "Jonathan M Davis" <jmdavisProg@gmx.com>:

> At some point, we're probably going to need to benchmark stuff more agressively and optimize Phobos in general more, because it's the standard library. And eliminating unnecessary memory allocations definitely goes along with that.
> 
> - Jonathan M Davis

On a related note, a while back I benchmarked the naive Phobos
approach to create a Windows API (wchar) string from a D
string with using alloca to convert the string on a piece of
stack memory like this: http://dpaste.1azy.net/b60d37d4
IIRC it was 13(!) times faster for ~100 chars of English text
and 5 times for some multi-byte characters.
I think this approach is too hackish for Phobos, but it
demonstrates that there is much room.

-- 
Marco

On 24 May 2013 14:11, Marco Leise <Marco.Leise@gmx.de> wrote:

> Am Thu, 23 May 2013 20:21:47 -0400
> schrieb "Jonathan M Davis" <jmdavisProg@gmx.com>:
>
> > At some point, we're probably going to need to
> > benchmark stuff more agressively and optimize Phobos in general more,
> because
> > it's the standard library. And eliminating unnecessary memory allocations definitely goes along with that.
> >
> > - Jonathan M Davis
>
> On a related note, a while back I benchmarked the naive Phobos
> approach to create a Windows API (wchar) string from a D
> string with using alloca to convert the string on a piece of
> stack memory like this: http://dpaste.1azy.net/b60d37d4
> IIRC it was 13(!) times faster for ~100 chars of English text
> and 5 times for some multi-byte characters.
> I think this approach is too hackish for Phobos, but it
> demonstrates that there is much room.
>

I don't think it's hack-ish at all, that's precisely what the stack is
there for. It would be awesome for people to use alloca in places that it
makes sense.
Especially in cases where the function is a leaf or leaf-stem (ie, if there
is no possibility of recursion), then using the stack should be encouraged.
For safety, obviously phobos should do something like:
  void[] buffer = bytes < reasonable_anticipated_buffer_size ?
alloca(bytes) : new void[bytes];

toStringz is a very common source of allocations. This alloca approach would be great in those cases, filenames in particular.