Thanks everyone for working on 2.068.1 which shipped on schedule.

retro:

We fixed a big amount of issues for 2.068.1, also some of the older
regressions (see http://dlang.org/changelog/2.068.1.html or
https://trello.com/b/XoFjxiqG/active).
Unfortunately fixing the template instantiation introduced 2 new
regressions with 2.068.1 (already fixed by now), one of which is severe
enough to require a 2.068.2 follow-up.

Both of the issues were present in 2.068.1-b2 and were found by testing
some of the most used projects on code.dlang.org. As the download
numbers (attached) show hardly anyone is actually participating in the beta.
Please test your code against the beta, it's the most important measure
to ensure a stable compiler.
I setup an automated jenkins test to cover at least some D projects
https://github.com/MartinNowak/project_tester. Please help by making
your open source code automatically testable and add it to the tester.

Planning:

Branching for 2.069.0 was planned in 2 weeks. Maybe we should delay that for another week for 2.068.2?

Version | Date
------------------|----------
Branch+Beta 2.069 | 2015-09-20
Release 2.069.0 | 2015-10-04

Please finish any work that is supposed to go into 2.069.0 within the next 2 weeks. We don't want to end up releasing half done features.

Tasks:

- [Rebench ddmd vs. dmd compiler speed. on Active](https://trello.com/c/OT6jlFNa/85-rebench-ddmd-vs-dmd-compiler-speed)

  We have to benchmark ddmd again to see whether it's fast enough.
  There is now gdc-4.9.3 and ldc-0.16.0-alpha2 which can build ddmd
(thanks Iain for backporting the C++ mangler). So we have a backup,  but
we'd have to integrate gdc or ldc with our build scripts and probably
still need compilers for OSX/Windows.

- [ptr data section](https://trello.com/c/Dsdxd1r3/56-ptr-data-section)

  I'm currently working on adding special GC root ptrdata sections, but
I ended up refactoring/rewriting hugs parts of the dt_t code, so I'm not
sure whether this will work out.

- [Finish SmartRef
Implementation](https://trello.com/c/pTlDuyBD/16-finish-smartref-implementation)

  Still WIP, been too busy with other stuff, but now my topmost priority.

- [@nogc tracehandler](https://trello.com/c/lbS0lUWc/86-nogc-tracehandler)

  We'll have to do that anyhow to fix [Issue 14993 – Allocating in a
destructor segfaults instead of throwing
InvalidMemoryOperationError](https://issues.dlang.org/show_bug.cgi?id=14993).

- [TTAS spinlock for
GC](https://trello.com/c/Xp44NwHG/87-ttas-spinlock-for-gc)

  There is already a finished PR waiting for the tracehandler to become
@nogc.

As usual please add your cards (or take some from the backlog) and add yourself as member when working on something.

-Martin

On Wednesday, 9 September 2015 at 17:14:34 UTC, Martin Nowak wrote:
> As the download
> numbers (attached) show

I found it much easier to read after rotating:

http://dump.thecybershadow.net/0802e4485377bca05c40159eacfbb890/beta_downloads.png

> hardly anyone is actually participating in the beta.

http://dlang.org/download.html should have links to Beta downloads during the testing period.

> - [Rebench ddmd vs. dmd compiler speed. on Active](https://trello.com/c/OT6jlFNa/85-rebench-ddmd-vs-dmd-compiler-speed)

I left some comments on PRs that caused changes on http://digger.k3.1azy.net/trend/ , maybe I should file issues for those. It seems like 90% of the time I mention something actionable in a merged PR, it gets ignored.

_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

Thanks, Martin!

Somewhat unrelated to this release, I have a question on this topic:

> Unfortunately fixing the template instantiation introduced 2 new
> regressions with 2.068.1 (already fixed by now), one of which is severe
> enough to require a 2.068.2 follow-up.

I haven't followed this very closely so please correct me if my perception is wrong. It seems that template instantiations has been fixed many times over the past many months (perhaps a year's time) and every time it was, new regressions were introduced that in turn needed more fixes.

I don't know how templates instantiation is currently implemented, but I can recognize the pattern because I've seen it (sadly) many times: it's the sign of a design that got to a catastrophic failure stage, in which the rate of defects outpaces the rate of patching and fixing them.

It seems to me it's time for a complete rethinking/redesign of template instantiation. A design that's comprehensible and robust and solves our perennial problems naturally (bloated object files, unnecessary work leading to slow compiles, linking issues, etc). What are the core compiler folks' thoughts on making this a priority?


Thanks,

Andrei
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 09/09/2015 08:15 PM, Andrei Alexandrescu wrote:
> I haven't followed this very closely so please correct me if my perception is wrong. It seems that template instantiations has been fixed many times over the past many months (perhaps a year's time) and every time it was, new regressions were introduced that in turn needed more fixes.

They always have been broken (e.g. multiple object files with -c), but
the real issues began with
https://github.com/D-Programming-Language/dmd/pull/2550 where we would
instantiate a lot less templates by making assumptions about templates
already being instantiated in a library.
This provided a significant speedup but comes with lots of subtle corner
cases.

> I don't know how templates instantiation is currently implemented, but I can recognize the pattern because I've seen it (sadly) many times: it's the sign of a design that got to a catastrophic failure stage, in which the rate of defects outpaces the rate of patching and fixing them.

The issues are getting smaller (edge edge cases) but we're overfitting
the implementation to the test cases instead of generalizing the
specification.
There is indeed way too much complexity buried in the template
implementation (maybe the design as well). This has always been the case
and I made a proposal to restucture the implementation
(https://github.com/D-Programming-Language/dmd/pull/4780#issuecomment-124087604).

> It seems to me it's time for a complete rethinking/redesign of template instantiation. A design that's comprehensible and robust and solves our perennial problems naturally (bloated object files, unnecessary work leading to slow compiles, linking issues, etc). What are the core compiler folks' thoughts on making this a priority?

It's definitely on the agenda but doesn't preclude working on the current implementation b/c it's unclear whether we can actually come up with a feasible and better solution.

-Martin

On 09/09/2015 07:33 PM, Vladimir Panteleev via dmd-internals wrote:
> On Wednesday, 9 September 2015 at 17:14:34 UTC, Martin Nowak wrote:
>> hardly anyone is actually participating in the beta.
> 
> http://dlang.org/download.html should have links to Beta downloads during the testing period.

https://github.com/D-Programming-Language/dlang.org/pull/1097

> I left some comments on PRs that caused changes on http://digger.k3.1azy.net/trend/ , maybe I should file issues for those. It seems like 90% of the time I mention something actionable in a merged PR, it gets ignored.

As you can see we're pretty busy already and github is a horrible
communication path (news/notification driven, spammy...).
Also you have to prioritize issues, it's really not that important when
the hello world size wiggles by a few kB.
Your Digger findings are really helpful, but it's necessary that we
prioritize bugs right away or the important ones drown in Bugzilla.
And this particularly does not mean to mark every other bug as
regression, b/c that's the only working communication channel we have
for high priority stuff.

On 09/09/2015 07:14 PM, Martin Nowak via dmd-internals wrote:
> Tasks:

- [(1) ship shared phobos lib for FBSD on Active |
Trello](https://trello.com/c/o4uzPZQk/88-ship-shared-phobos-lib-for-fbsd)
  I'd like to ship shared phobos libraries for FreeBSD with the next
release.

On Wednesday, 9 September 2015 at 19:20:47 UTC, Martin Nowak wrote:
>> I left some comments on PRs that caused changes on http://digger.k3.1azy.net/trend/ , maybe I should file issues for those. It seems like 90% of the time I mention something actionable in a merged PR, it gets ignored.
>
> As you can see we're pretty busy already and github is a horrible
> communication path (news/notification driven, spammy...).

It was an observation, not a complaint.

> Also you have to prioritize issues, it's really not that important when
> the hello world size wiggles by a few kB.

Issues that are not important enough to fix can be closed as WONTFIX, as long as someone in the position to make that decision has a look at it. I really don't think this is a problem.

> Your Digger findings are really helpful, but it's necessary that we
> prioritize bugs right away or the important ones drown in Bugzilla.
> And this particularly does not mean to mark every other bug as
> regression,

That's just cold. So do you want me to stop checking if filed bugs are regressions? That doesn't seem logical to me.

Many people seem to not realize that the bugs they run into are actually regressions and their code worked fine in older versions of D, but I do not think that these are less important to fix. A regression is a regression, no matter how we discover it - not all people report bugs they run into, and if we ignore some regressions, they might affect other users.

> b/c that's the only working communication channel we have
> for high priority stuff.

Bugzilla has a priority field, but it doesn't seem to be used much.

_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

I'm very much in favor of seeing shared libraries become a reality, but there's a necessary process that's needed to ensure the required behavior: backwards abi compatibility.  The major projects that ship shared libraries (ie, libgcc, libc, etc) have tooling (that I've never looked at) to detect breakage.  This covers both no changes to existing symbols as well as controlled and acknowledged introduction of new symbols.  Production quality shared library support is a big step up in responsibility.


On 9/9/15 12:32 PM, Martin Nowak via dmd-internals wrote:
> On 09/09/2015 07:14 PM, Martin Nowak via dmd-internals wrote:
>> Tasks:
>
> - [(1) ship shared phobos lib for FBSD on Active |
> Trello](https://trello.com/c/o4uzPZQk/88-ship-shared-phobos-lib-for-fbsd)
>    I'd like to ship shared phobos libraries for FreeBSD with the next
> release.
>
>
>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals@puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 09/09/2015 03:20 PM, Martin Nowak via dmd-internals wrote:
> As you can see we're pretty busy already and github is a horrible
> communication path (news/notification driven, spammy...).

Yeah, it's killing me too!

> Also you have to prioritize issues, it's really not that important when
> the hello world size wiggles by a few kB.

That sounds reasonable but I must disagree. For one thing, the wiggling is going a lot more often on the increase than on the decrease. There is one way to reduce code bloat, and that is measure it and make it easier for it to go down than go up. Then you have established a ratchet mechanism which will ineluctably push down on bloat. Letting things loose means, well, that'll never happen.

On the HHVM team the managers have researched and exploited with immense success this ratchet pattern. That team's lore includes anecdotes from wildly different domains all telling the same story.

Of obese people who lose weight, 80% gain it back. Who doesn't? Those who weigh themselves each day and COMPLETELY FREAK OUT when they see half a pound over their target weight. They do ANYTHING IT TAKES to take their weight back to where it was.

This story is a legend among the HHVM team because we applied it as well: http://jamesclear.com/marginal-gains

That's how we got HHVM from 5x slower to 4x faster than the baseline: by working on every percent. And by not letting things wiggle.


Andrei
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 09/09/2015 09:50 PM, Andrei Alexandrescu wrote:
> That sounds reasonable but I must disagree. For one thing, the wiggling is going a lot more often on the increase than on the decrease.

We know very well where the bloat comes from (ModuleInfos, class
references, eager template analysis) and unless someone tackles those
bigger issues, the small stuff really doesn't matter that much.
In the meantime we're reverting useful code b/c hello world must not
grow
https://github.com/D-Programming-Language/phobos/pull/2956#issuecomment-128717113.