D for Game Development (page 13)

August 09, 2015

Re: D for Game Development

Posted by Manu
in reply to Walter Bright

Permalink

Manu

Posted in reply to Walter Bright

Permalink

On 9 August 2015 at 15:31, Walter Bright via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 8/8/2015 7:40 PM, Manu via Digitalmars-d wrote:
>>
>> 1. DMD has unsatisfactory codegen for anything other than debug builds.
>
>
> Do you mean the codegen is slower? But consider that the bottleneck in most programs is a small section of code. Taking a good look at the generated code for that and comparing with another compiler can often hint at an easy improvement to dmd that can address that bottleneck.
>
> But waiting for someone else to discover the same thing on some other piece of code means you'll be waiting a long time.

I understand, but that's not sustainable.
We can't be in a situation where I'm at work with a deadline, and I
need to invoke the community to address a codegen issue that's
blocking a release. Unexpected surprises like that with unknown/long
turn-around times aren't a practical commitment.
Last time I was concerned with this, x87 was destroying DMD
performance. Floats were x87, moved to XMM (according to x64 ABI) when
passed to functions, then moved back into x87 on the other side when
they're operated on.
Float code was overwhelmed with shuffling values between registers.
Basically all hot code is float code in my line of work.

I think Ethan was in discussions with you about this some time back? It was a problem at the time I left Remedy.

Also, I think one of the big issues is the optimiser in general. The
inliner's just not fantastic, and D depends on it perhaps even more
than C++, especially when using the modern lazy code style.
I'm sure I can hand-assist DMD to produce the assembly I expect for
some hot function with some work, but traditional hand-assisted code
generation all falls apart when you start stacking lazy-ranges and
generally using the modern D style.
DMD's backend has almost no development, it's advancing at near-zero
velocity compared to LLVM. We need to get out of the situation where
LDC and GDC are perpetually 1-2 versions behind.

Incidentally, In my recent (dead-end) experience with LDC, I get endless ICE's whenever I try to do anything mildly complex. When you have your Debug and Release build straddling language versions and compilers with different ICE characteristics, it's just not a practical workflow. Only green-fields projects or the simplest of simple apps can hope not to trigger bugs constantly in that environment.

>> 2. DMD generates x87 code, and uses real everywhere.
>
>
> Less so now than it used to. For float and double, it uses SIMD.

Okay, is this fairly recent? My last set of comprehensive tests are probably out of date.

>> We can't be
>> generating new x87+real instructions in 2015. It's deprecated
>> hardware!
>
>
> x87 works on every x86 CPU, and I doubt it will ever go away, deprecated or not. Why was it a problem for you?

Massive register swapping was the issue. I'd like to think register swapping should never be implicitly generated unless I *explicitly* type the real keyword, I don't want to see x87 opcodes.

>> 2. LDC has a lot more bugs than DMD (which still has quite a few
>> bugs), and the Windows build is new and even more immature still.
>> 3. LDC has no debuginfo. **biggest practical issue by far!**
>> 4. LDC/GDC are always a few versions behind DMD. This creates an
>> awkward/almost-impossible situation when you rely on DMD to build
>> debug code, and LDC to build releasable code. Depending on 2 flaky
>> compilers is even less fun than one.
>> 5. DLL's were really flakey at the time, I'm not sure how they are
>> now, but I'm concerned they may not be much better.
>> 6. Shared druntime; we were loading D code from DLL's, lots of them.
>> It's not really reasonable for each DLL to have it's own druntime
>> instance, they need to share one provided by the host app.
>
>
> I agree, and now we ship a Phobos DLL, resolving that issue.

Really? Where is it? (I can't see it in the distribution).

>> 7. D binaries are *gigantic*, I don't know why. D seems to link the world, and I'm certain that most of the crap that's linked is never referenced or executed... it just makes gigantic binaries for some reason, even with line-for-line translations of C/C++ code. That's not real good on games consoles where icache is king.
>
>
> Yes, that's an ongoing problem. I need to spend some time on that one.

That'll be nice. Bear in mind, I'm just giving an account here of why we were unable to produce blog posts and hype from our experience using D at Remedy, at least up until the time I left, I'm not sure where they are with it now. I hope it didn't fizzle, but there were more problems than we initially hoped for, and I think that damaged confidence a lot.

In the short term? I struggle to imagine the ddmd transition taking less than 3-6 months.

Personally, if I were to pin a #1 ticket related to my 2015 workload, it would be Emscripten support, and LDC as default D compiler ;)

>> I tried to use D in my current workplace on 3 occasions now, and ran
>> into a different set of problems.
>> In the first attempt I tried to write a simple vibe.d webserver app,
>> but it kept crashing randomly when I was using the API's exactly as
>> intended (apparently a client was posting `/n` instead of `/r/n`, or
>> maybe the inverse of that, and it didn't like it).
>
>
> Were these bugs reported?

Yup, and some of them fixed. Other's remain because I couldn't create
a reduced test environment; async communications using websockets in
this case.
At the time I was actually heckled by people here telling me I was
stupid to try and use vibe.d commercially.
I was fairly surprised, it seems like one of the D community's biggest
success stories, and people were (almost violently) telling me I'm
stupid to attempt to use it in production.

>> In my second attempt, progress was interrupted by some ICE's, linking with some C++ code caused various problems.
>
>
> ICE's in what? What problems?

LDC. I think it's the most high-priority tool in the ecosystem, and it doesn't have anywhere near enough manpower. I am 100% in favour of the many calls for LDC as front-line compiler.

Might I suggest - and I'm sure you won't like this, but I think it would do worlds of good; your experience implementing MS debuginfo in DMD is invaluable and fairly unique. If you could contribute that debuginfo support to LLVM, it would make a world of difference.

>> A final recent attempt was basically a non-starter on account of no Qt bindings that work &&/|| are up to date.
>
>
> Bindings aren't too hard to update.

Qt is not a small library. Binding Qt appears to be a much, much bigger task than my task. I'd need to take it as an ongoing hobby project, but I'm overrun with those already >_<

>> I'm getting pretty close to calling it a day. Is 6 years enough time
>> to repeatedly prove to myself that I'm chasing a fantasy?
>> People keep talking about Rust, I'm thinking of giving it a shot.
>>
>> I think D needs a budget, and paid programmers employed to do boring
>> shit that makes the whole thing more stable and robust. Mozilla seems
>> to have this(?)
>> D seems to consistently offer 95% solutions to problems. 95% is not a
>> solution, it's just another problem.
>>
>> Oops, I ranted. Sorry! :)
>
>
> I think what stalled Remedy's use of D is you left Remedy. Note that while you were there I went to Finland largely at my own expense to help make it work for Remedy. After you left, there wasn't much interest from Remedy.

You're probably right to an extent, I was really pushing for it and I had the latitude to do so at the time, but interest was actually very high. Most commercial dev's I've worked with tend not to engage in the open-source space though, so maybe it just wasn't part of their habit to involve in communities like this one. I really do appreciate the effort you made. I hope it was valuable development regardless, I think a lot of important things came form it.

The reality is that it did take a lot longer to get to a workable
state than we had initially budgeted though. Off the back of that, I
think there was a lot of confidence lost, and we also entering a phase
of crunch.
The gravity of crunch in the video games industry can't be
understated; once the crunch coding begins, there is very little time
in anyone's life to get some new experimental tech across the line. We
reached the point where we were able to implement systems in D and it
was looking great, but by that time, none of the systems we planned to
implement in D were written in D because their deadlines had already
passed, so they fell back to C instead, and momentum was lost.

> I note that I rarely know just what is blocking you. For example, you say above you got some ICEs. What ICEs? We cannot fix bugs we don't know about.

LDC bugs, I reported them all. I couldn't sufficiently reduce test cases in most cases though. The latest release has some promising looking bugs fixed and I'm keen to try it out again on that code, but the time for that code has passed so it's just another opportunity missed at this point. I'll try it out enthusiastically, but what are the odds I'm blocked again a couple of hours later by some other minor and frustrating thing?

> I would also very much appreciate if you would pitch in to help with things that are blocking you, like updated Qt bindings. Generally, people around here tend to contribute fixes for things that personally interest or block them. Priorities for you aren't priorities for others.

I understand, and I've come to terms with this. But the practical
effect of that is sadly just to distance me from D. And I'm honestly
not trying to be negative, just realistic.
I looked into writing Qt bindings, but it's a pretty big job, probably
much bigger than the job they are to be used for. The question I have
to ask though is, would that enable me to do my work, or would I just
run into the next problem? That was the blocker, but there may have
been endless problems after that.
I can't reasonably gamble with my deadlines in that way. It's a bit of
a chicken-and-egg problem.

Sadly, I'm working more overtime than ever this year, so I just don't
have after-hours at the moment. You might have noticed my relative
silence.
I'm still trying to polish up std.color (which was a blocker for one
of my after-hours projects) in the free time I have. It's kinda
sitting in limbo waiting on ctfe float ^^ operator, and also
reviews...

> D doesn't have a community so large that you can sit back and assume many others will run into the same issues you have and will fix them. You have to be more proactive.

I understand, but I don't have time to allocate.
The only practical suggestion that comes to mind is that D needs a
budget, and employees. People paid to take the really boring stuff
seriously, and ideally prioritise paying corporate users, such that
they can have confidence that their issues won't be show-stoppers.
That probably wouldn't work well in practise though. While the number
of bugs and issues in DMD are steadily declining, I'm finding the
show-stoppers are just moving elsewhere.
LDC has ICE's in the quantity that DMD used to have, Android and iOS
support (ie, most computers in the world) are moving along rather
slowly; they are single-man teams and don't have core support.
Emscripten, NaCl, etc aren't supported.
Almost all my recent work can't ignore these platforms. Bugs relating
to external blockers like these don't belong in the DMD database.

In fact, we've been discussing for a few months that we'd have have another very promising opportunity to use D at work in a really appropriate context if I could rely on Android and iOS appearing within 6-12 months or so. It would take a plug-in form like at Remedy, which I think is a really practical transition path for a company with engineers that need opportunities to get their feet wet, but can't afford to bet the farm.

> Looking at open bugzilla issues you've commented on:
>
> https://issues.dlang.org/show_bug.cgi?id=5227 https://issues.dlang.org/show_bug.cgi?id=6343 https://issues.dlang.org/show_bug.cgi?id=8845 https://issues.dlang.org/show_bug.cgi?id=9065 https://issues.dlang.org/show_bug.cgi?id=9137 https://issues.dlang.org/show_bug.cgi?id=10203 https://issues.dlang.org/show_bug.cgi?id=10959 https://issues.dlang.org/show_bug.cgi?id=11541 https://issues.dlang.org/show_bug.cgi?id=12440 https://issues.dlang.org/show_bug.cgi?id=13265 https://issues.dlang.org/show_bug.cgi?id=14260
>
> That's it. None of them seem to be ones you brought up in your post.

This small number actually represents how far DMD has come recently. I recognise and appreciate how far the compiler has come; I don't have codegen or language issues very often at all these days. And likewise, they haven't been the focus of my complaints. LDC not supporting debuginfo for instance doesn't appear on the DMD bugzilla.

Note, x^^y is still in that list. That's blocking std.color ;)

> To get attention for issues that are blocking you you absolutely have to file bugzilla reports. That doesn't guarantee they will be fixed, but if you do not file them, it DOES guarantee that they will NOT be addressed, and you'll get frustrated.

I think you misunderstood the point of my rant. What I was trying to share is past experience, not the state of the ecosystem today. It's important to note the recurring experience where there always seems to be some random thing that blocks me. I've approached many different kinds of projects now, from a lot of different angles. I'm still unable to make one stick. I'm obviously doing something wrong repeatedly, but you can't say I haven't given it a jolly good effort!

I'm pretty consistent at logging bugs, except those that I can't justify burning a few hundred dollars of work time to produce a reduction that I could log are often lost (and often quite important). I'm often inhibited by the fact that I attempt to use D at work, and offices are stupid proprietary places so I can't share code >_<

My job and workloads have changed a lot recently, so I'm facing new problems.
But yeah, LDC and missing libraries+platforms are the source of almost
all my issues in 2015. It would be great for DMD devs to be building
LDC routinely, and this situation where GDC and LDC are a couple of
versions behind DMD has to end asap.

On 9 August 2015 at 20:04, Dmitry Olshansky via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 09-Aug-2015 12:33, Manu via Digitalmars-d wrote:
>>
>> On 9 August 2015 at 13:34, Rikki Cattermole via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>>>
>>> On 9/08/2015 2:40 p.m., Manu via Digitalmars-d wrote:
>>>>
>>>>
> [snip]
>>>
>>> Even though it is hard for you to still be here, I hope you do continue
>>> to
>>> help D grow. This is a good community, we just need time before real
>>> adoption. E.g. we are only just now starting to unify projects.
>>
>>
>> Cheers, but don't get me wrong; I've barely contributed anything of value other than advocacy and little bits and pieces here and there. I don't deserve any credit, I just complain about things that make it impossible for me to get on with my work.
>
>
> std.simd though would be awesome to have ;)

Yeah, I keep coming back to it, and getting myself stuck with various
troubles making it work good. I'm perpetually unhappy with my code,
and I just need a really good block of time to get through it.
If people want to review std.color and help get that off my
short-list, I promise I'll make std.simd my top priority :P

I also haven't had need for it in my recent work, so it's not being motivated by necessity since I haven't been game-dev-ing much.

Forums