With the 2.068.0 out the door let's look ahead and plan the next 2 weeks.

Small retro:

Lot's of momentum in the final release phase just look at our done board (https://trello.com/b/XoFjxiqG/active, cards will get archived in 2 days). Thanks to the many people helping out.

Let's keep it that way for the next release, though we should spare the initial inertia (the release took more than 4 weeks).

As a reminder 2.068.1 is due in 1 month, 2.069.0 in 2 month.
I'm aiming for a 1 week beta period for point releases and 2 weeks for
full releases, i.e. 3 weeks until 2.068.1-b1 and 6 weeks until 2.069.0-b1.

Planning:

There are a few leftover things from the release phase that we should handle before jumping on ddmd work.

- Dropping -property waits for an offical OK from Walter
  https://trello.com/c/1dQh4gxm/35-drop-property

- Debug druntime tests wait for someone to look at FBSD
  shared library guts
  https://trello.com/c/zmsotPH2/20-fix-debug-druntime-tests-for-fbsd

- There is some pending work on VS2015 support
  https://trello.com/c/5OWcNIGy/33-vs2015-support

- m32coff targets are here and simply need to be wired
  with the release scripts
  https://trello.com/c/al6RlkJP/44-mscoff32-libs-for-release

- We hackfixed a mangling issue for mixins and lambdas
  and need to come up with a proper solution.

https://trello.com/c/YuW4JycJ/45-mangling-of-mixins-and-lambdas-can-change-w-unittest-or-other-versions

- I noticed a few missing things while documenting the rangified functions.
  https://trello.com/c/vC7GfbPA/46-follow-up-on-rangified-functions

- I want to finish my WIP smart refs which didn't make it for 2.068.0
  https://trello.com/c/pTlDuyBD/16-finish-smartref-implementation

- The newly added compiler internal helper functions should get their own module or so instead of being appended to object.d

https://trello.com/c/Vf4Gasoa/47-follow-up-on-compiler-internal-helper-module-and-arraydtor-error-messages


DDMD:

Regarding the ddmd switch I'd suggest to first make a spec story to understand and break down what needs to be done. We should closely coordinate this with people from the GDC and LDC team.

https://trello.com/c/4NtxWDtK/48-ddmd-bootstraping-and-backwards-compatibility-guarantees

Discussing this on dmd-internals seems like a good idea to me.

Then we can schedule whatever tasks fall out of the discussion for this and the next sprint.


That already sound like a lot for the next 2 weeks, though some of the phobos devs might still want to add a few stories or groom the backlog, e.g. https://trello.com/c/jtmRtrHD/29-fix-binary-size-for-tempfile.

-Martin

On 10 August 2015 at 18:30, Martin Nowak via dmd-internals < dmd-internals@puremagic.com> wrote:

> With the 2.068.0 out the door let's look ahead and plan the next 2 weeks.
>
> Small retro:
>
> Lot's of momentum in the final release phase just look at our done board (https://trello.com/b/XoFjxiqG/active, cards will get archived in 2 days). Thanks to the many people helping out.
>
> Let's keep it that way for the next release, though we should spare the initial inertia (the release took more than 4 weeks).
>
> As a reminder 2.068.1 is due in 1 month, 2.069.0 in 2 month.
> I'm aiming for a 1 week beta period for point releases and 2 weeks for
> full releases, i.e. 3 weeks until 2.068.1-b1 and 6 weeks until 2.069.0-b1.
>
> Planning:
>
> There are a few leftover things from the release phase that we should handle before jumping on ddmd work.
>
> - Dropping -property waits for an offical OK from Walter
>   https://trello.com/c/1dQh4gxm/35-drop-property
>
> - Debug druntime tests wait for someone to look at FBSD
>   shared library guts
>   https://trello.com/c/zmsotPH2/20-fix-debug-druntime-tests-for-fbsd
>
> - There is some pending work on VS2015 support
>   https://trello.com/c/5OWcNIGy/33-vs2015-support
>
> - m32coff targets are here and simply need to be wired
>   with the release scripts
>   https://trello.com/c/al6RlkJP/44-mscoff32-libs-for-release
>
> - We hackfixed a mangling issue for mixins and lambdas
>   and need to come up with a proper solution.
>
>
> https://trello.com/c/YuW4JycJ/45-mangling-of-mixins-and-lambdas-can-change-w-unittest-or-other-versions
>
> - I noticed a few missing things while documenting the rangified functions.
>   https://trello.com/c/vC7GfbPA/46-follow-up-on-rangified-functions
>
> - I want to finish my WIP smart refs which didn't make it for 2.068.0
>   https://trello.com/c/pTlDuyBD/16-finish-smartref-implementation
>
> - The newly added compiler internal helper functions should get their own module or so instead of being appended to object.d
>
>
> https://trello.com/c/Vf4Gasoa/47-follow-up-on-compiler-internal-helper-module-and-arraydtor-error-messages
>
>
> DDMD:
>
> Regarding the ddmd switch I'd suggest to first make a spec story to understand and break down what needs to be done. We should closely coordinate this with people from the GDC and LDC team.
>
>
> https://trello.com/c/4NtxWDtK/48-ddmd-bootstraping-and-backwards-compatibility-guarantees
>
> Discussing this on dmd-internals seems like a good idea to me.
>
> Then we can schedule whatever tasks fall out of the discussion for this and the next sprint.
>
>
- Do we need to keep ddmd working with 2.068?

For the time being, yes, and for as long as possible.

I'd like to see continual bug fix releases incase anything slipped into 2.067 or 2.068 that broke GDC or LDC's any part of their interface, codegen or strictly-typed backend.  (See for instance, PRs 4062, 4075, 4077, 4212, and probably many more just to get 2.066 working).


- When will gdc and ldc catch up and switch to a self-hosted frontend.

No set time on my side.  Currently I'm doing what is effectively a 70-80% rewrite of all codegen.  Eg: This is for the complete removal of the IRState definition from our glue.

https://github.com/D-Programming-GDC/GDC/commit/f7ad1ef43021e95eb4af387fa1cf300f9da856b9

I've written up a wiki (both myself and Johannes are crossing things off as
we complete them):

http://wiki.dlang.org/GDC/CurrentReleaseTasks


- Any bootstrap strategies for cross compiling?

Initially, use 2.068 as baseline, then bootstrap with the later versions (written in D).  This could be integrated into the GCC build without too much hassle, as they themselves have a three-stage build+bootstrap process.

I'm admittedly worried that this will block any work I'm doing with upstream GCC.  In my most ideal case, I'd want 2.068 merged into the GCC ecosystem, then have it trickled down into all major vendors in whatever major release we end up being shipped in before switching.  This gives distributors the best opportunity to manage building the self-hosted sources without running into an impossible Catch-22 situation.

Debian/Ubuntu are currently in the best position as the maintainer has kindly integrated GDC into the GCC binaries (albeit with favours):

https://packages.debian.org/sid/gdc-5

I would like to see libphobos (or at least libdruntime) being made
available for more targets though.

Current list is: [amd64, armel, armhf, i386, x32]


- How to test compatibility?

Compatibility with what?  ARM? SPARC? PPC?  Communicating across C++ <-> D?


Regards
Iain.

On Monday, 10 August 2015 at 16:30:37 UTC, Martin Nowak wrote:
> With the 2.068.0 out the door let's look ahead and plan the next 2 weeks.
>
> There are a few leftover things from the release phase that we should handle before jumping on ddmd work.

I think the remained regression should be fixed.

- fix compilation speed
https://trello.com/c/L0nV131G/17-investigate-fix-compiler-slowdown

- Kenji
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 10 Aug 2015, at 18:30, Martin Nowak via dmd-internals wrote:
> With the 2.068.0 out the door let's look ahead and plan the next 2 weeks.

I'd appreciate if we could have another look at struct lifetime handling. The whole argprefix business introduced in 2.067 made life more difficult for LDC, and does not seem to be anywhere near complete: https://issues.dlang.org/show_bug.cgi?id=14903

Since the template emission is still being discussed, I'd also like to draw attention to the following regression: https://issues.dlang.org/show_bug.cgi?id=14901

As you can probably guess from my GitHub feed and bug reports, I'm currently spending most of my D time on making DMD/LDC 2.067 work for one of our corporate users. Thus, I can't really make any big picture plans as far as I am concerned. Focusing on DDMD exclusively for 2.069 seems like a workable plan, although I'd really like to see some of the long overdue big issues (e.g. 314) fixed soon. The amount of unintended regressions those cause in big code bases on every release is staggering. Just about *every* fix for them, no matter how disruptive they might seem, would still be a giant step over having to deal with related fallout on every single compiler update. Backwards compatibility for new compiler releases is currently an illusion when it comes to even just moderately sized projects.

 — David
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 8/10/2015 5:35 PM, David Nadlinger via dmd-internals wrote:
> On 10 Aug 2015, at 18:30, Martin Nowak via dmd-internals wrote:
>> With the 2.068.0 out the door let's look ahead and plan the next 2 weeks.
>
> I'd appreciate if we could have another look at struct lifetime handling. The whole argprefix business introduced in 2.067 made life more difficult for LDC, and does not seem to be anywhere near complete: https://issues.dlang.org/show_bug.cgi?id=14903
>
> Since the template emission is still being discussed, I'd also like to draw attention to the following regression: https://issues.dlang.org/show_bug.cgi?id=14901
>
> As you can probably guess from my GitHub feed and bug reports, I'm currently spending most of my D time on making DMD/LDC 2.067 work for one of our corporate users. Thus, I can't really make any big picture plans as far as I am concerned. Focusing on DDMD exclusively for 2.069 seems like a workable plan, although I'd really like to see some of the long overdue big issues (e.g. 314) fixed soon. The amount of unintended regressions those cause in big code bases on every release is staggering. Just about *every* fix for them, no matter how disruptive they might seem, would still be a giant step over having to deal with related fallout on every single compiler update. Backwards compatibility for new compiler releases is currently an illusion when it comes to even just moderately sized projects.

There's always going to be reasons not to switch to ddmd. We need to just do it.

I don't know what you mean by 314 causing regressions in every release?
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 08/10/2015 07:10 PM, Iain Buclaw wrote:
> - Do we need to keep ddmd working with 2.068?
> 
> For the time being, yes, and for as long as possible.

Sticking to an old version of the compiler defeats a major reason to switch to self-hosting. By using D ourselves for a big project we'll resolve particular issues that'll come up. But if we cannot profit from solving those, this will have little impact.

So what is the major reason we need backwards compatibility for more
than 1 version?
After all you can always do the following.

download2068
foreach (ver; 2.069 .. 2.085)
{
    git checkout ver
    buildWithPreviousVer
}

It seems to me the key problem is to guarantee that the above works,
which is what I meant with testing compatibility.
The minimum IMO would be the classical self-compiling check, and of
course a rigorous test suite.

buildWithPreviousVer
test
buildWithCurrentVer
test
buildWithCurrentVer <- binary must be equal to buildWithCurrentVer above
test

On 08/10/2015 07:13 PM, Kenji Hara via dmd-internals wrote:
> 
> I think the remained regression should be fixed.
> 
> - fix compilation speed https://trello.com/c/L0nV131G/17-investigate-fix-compiler-slowdown
> 
> - Kenji

I agree, it would be great to fix this, but only if we can control the
risk. We can't do a major change of an essential component while
transitioning to ddmd.
Also there are plans to refactor/reimplement the template instantiation,
so anything more than a fix would make more sense after that.

On 11 August 2015 at 13:05, Martin Nowak via dmd-internals < dmd-internals@puremagic.com> wrote:

> On 08/10/2015 07:10 PM, Iain Buclaw wrote:
> > - Do we need to keep ddmd working with 2.068?
> >
> > For the time being, yes, and for as long as possible.
>
> Sticking to an old version of the compiler defeats a major reason to switch to self-hosting. By using D ourselves for a big project we'll resolve particular issues that'll come up. But if we cannot profit from solving those, this will have little impact.
>
>
IMO, it only enforces that we don't immediately jump onto using new features introduced post-2.068 within the compiler so quickly.



> So what is the major reason we need backwards compatibility for more
> than 1 version?
> After all you can always do the following.
>
> download2068
> foreach (ver; 2.069 .. 2.085)
> {
>     git checkout ver
>     buildWithPreviousVer
> }
>
>
Because I operate around GCC releases, rather than DMD.  In the 6-8 months that a GCC development cycle and release is done, there could be many DMD releases during that time.  If (the nonexistent) GDC-6.2 built on 2.068 cannot build GDC-7.1 (2.072) or current GDC-8.0 development snapshot (2.075), then I have a serious problem with upstream development.



> It seems to me the key problem is to guarantee that the above works,
> which is what I meant with testing compatibility.
> The minimum IMO would be the classical self-compiling check, and of
> course a rigorous test suite.
>
> buildWithPreviousVer
> test
> buildWithCurrentVer
> test
> buildWithCurrentVer <- binary must be equal to buildWithCurrentVer above
> test
>
>
This cycle is what's done with GCC build process, if you change a few words around.

buildWithHostCompiler
bootstrap
buildWithBuildCompiler
bootstrap
buildWithBuildCompiler
Compare md5sum of second and third stage *.o files


Regards
Iain

On 11 Aug 2015, at 11:25, Walter Bright via dmd-internals wrote:
> There's always going to be reasons not to switch to ddmd. We need to just do it.

I do fully agree. For exactly this reason, I worked with Daniel to get DDMD built by LDC up and running the day after DConf instead of immediately going on my trip through Utah as originally planned. :)

> I don't know what you mean by 314 causing regressions in every release?

Every time somebody adds/removes imports in Phobos or makes them function-local, it has a chance of causing breakage in client code because of the various holes in the module system. I've witnessed this with two releases in the Weka codebase now. With that observation, I was just trying to make the point that we should not shy away from fixing those issues due to fear of causing unforeseen regressions, because we are already doing so all the time.

 — David
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 08/10/2015 06:30 PM, Martin Nowak via dmd-internals wrote:
> Regarding the ddmd switch I'd suggest to first make a spec story to understand and break down what needs to be done. We should closely coordinate this with people from the GDC and LDC team.

From an email conversation with Walter and Daniel:

On 08/11/2015 10:11 AM, Walter Bright wrote:
>
>
> On 8/11/2015 12:41 AM, Daniel Murphy wrote:
>> You need to sign off on the generated D source,
>
> It passes the test suite, right? Then let's just switch to it. I expect we'll fix it up over time, after all, that's the point.
>
>>   and we need to pick
>> host compilers for each major platforms that produce a fast enough dmd
>> binary.
>
> Let's use 2.068 dmd for that. When ldc/gdc are up to 2.068, we can switch to them if necessary.
>

Please don't underestimate the problem.

If we release a self-hosted compiler that is 30% slower, then the
message between the lines is that's b/c of D is slower than C++.
And we already had some major slowdowns in the past versions, adding
another 30% risks to kill one of D's major selling points (compile times
have been stressed a lot as major advantage but aren't nearly as fast as
they used to be).

I just took these numbers from dmd/ddmd master (ddmd was compiled with
-O -inline -release).

relative numbers:
https://dawg.eu/dmd_vs_ddmd_time.html
absolute numbers:
https://dawg.eu/dmd_vs_ddmd_time.html#abs

And the profiles of both are very similar, e.g. comparing the relative
times spend compiling dub.
https://dawg.eu/dmd_vs_ddmd_prof.svg

The _d_new* and _memcpy times in ddmd are compensated by malloc time in dmd, so the only real difference is TemplateInstance::semantic which is only good for 1%.

A uniformely slowdown of 30% b/c of an inferior backend isn't something we can resolve. So the necessity for ldc/gdc is already clear.