Thread overview
[dmd-internals] 9th Sprint Planning
Nov 28, 2015
Martin Nowak
Nov 28, 2015
Iain Buclaw
Nov 28, 2015
Martin Nowak
November 28, 2015
Long overdue, welcome to the 9th sprint planning.

4 weeks means there is a lot to talk about, so please read the somewhat lengthy mail and participate in the planning, it's well worth your time.

After the unplanned 2.069.1 point release we'll now start the beta for
the 2.069.2 bugfix release which is scheduled to be released on 3rd of
december.
As we don't have much regression reports the release should be very
small. We're still urgently searching for someone to include
std.experimental.allocator in phobos' windows makefiles.
https://trello.com/c/thMHmnvz/133-issue-15281-std-experimental-allocator-package-d-not-included-in-build-script


##### Post-mortem of Windows installer bug #####

During the update of the NSIS installer to support VS2015 a bug was
introduced that caused an infinite loop (it was caused by a misplaced
label). It took us a few days to notice the multiple forum comments.
It then took a few more days to get someone affected by this bug to help
us, b/c we were unable to reproduce the issue.
After quite some trial and error and spending more than 1 hour to build
a NSIS installer with logging, we finally managed to pinpoint the bug
and Rainer made a fix.

Last time I looked at the histogram, about 50% of our users download the Window installer. Because of this severity we decided to immediately follow up with a point release.

There are a few learning from that.

1.) We don't systematically test our installers. For the Windows installer it would be installing w/ and w/o existing dmd installation times w/ and w/o existing VS installation. We really need some volunteers testing those 4 combinations.

2.) Using NSIS is a major pain. The platform hasn't been updated since 6
years (though suddenly there is a beta for NSIS 3.0).

I have to host the unzip plugin binary myself before it vanishes from the internet, we need to use a "special" NSIS build that's able to deal with "large" strings (more than 1KB). https://dlang.dawg.eu/downloads/others/ http://nsis.sourceforge.net/Nsisunz_plug-in http://nsis.sourceforge.net/Special_Builds#Large_strings

But the worst thing is that the installer script is written in a sort of
macro assembler, which is the actual cause for the bug.
https://github.com/D-Programming-Language/installer/blob/cc4d45a4cdde945ea2cd4eeb3dd05b4289f786ee/windows/d2-installer.nsi

3.) It would be better to integrate with Windows Installer and use a .msi installer.

https://en.wikipedia.org/wiki/Windows_Installer

We got a friendly recommendation to use InnoSetup for nicer .msi installers. I'm not yet willing to spend the effort to rewrite the installer, but on the next occasion (bug) we should do it.


##### Review of last sprint #####

I haven't followed the upstream development too closely, so maybe
someone else can fill in more details.
I just want to use the occasion to highlight how spending some thought
and fixing an issue properly, instead of adding yet another hack on top
of an already messy implementation, can profoundly improve our code base
and subsequently allow to more easily implement other improvements.

https://github.com/D-Programming-Language/druntime/pull/1419#issuecomment-152548521 https://github.com/D-Programming-Language/druntime/pull/1421 https://github.com/D-Programming-Language/druntime/pull/1422 https://github.com/D-Programming-Language/druntime/pull/1110

I spend most of the past 4 weeks with infrastructure work.

## dlang-bot

As you may have noticed already there is now a dlang-bot on github posting bugzilla links on PRs, so that people no longer have to do this this manually and can spend their time on useful PR descriptions instead.

https://github.com/dlang-bot?tab=activity

This was also good for a small article (w/ a problematic code/words ratio), that was read by quite a few people. https://code.dawg.eu/writing-a-vibed-app.html

I'm planning on another vibe.d article about a git based wiki app I'm already using since a few years.

## nightlies and installer scripts

I spend a lot of time to setup nightly builds. First I was using my
local dev machine with a rtc wakeup and cron job, but that turned out to
be too unreliable (kernel/package updates, S3 wakeup hang, forgot to
unlock gpg key). So I now migrated this to a dedicated server (a low end
box that's 4x slower than my dev machine) and hope to stabilize and
publish everything over the next few days.
You can find a beta of the nightlies service here.

https://builds.dawg.eu/

If you want to install a nightly you might try out our new install script (and yes that safe to pipe to bash [¹]).

curl -fsSL https://builds.dawg.eu/install.sh | bash -s dmd-nightly

If you don't like that, do something else, e.g.

curl -fsSLO https://builds.dawg.eu/install.sh
curl -fsSLO https://builds.dawg.eu/install.sh.sig
gpg --verify install.sh.sig
vim install.sh
chmod +x install.sh
./install.sh dmd-nightly

or just download the latest nightly directly.

https://builds.dawg.eu/dmd-nightly/dmd.master.linux.tar.xz

The installer script with create a ~/dlang folder, installs itself, can
install almost any version of dmd/gdc/ldc, is based on the travis-ci and
heroku script and should be pretty self-explanatory.
If you find something non-obvious, please help to improve it [²].

I hope to find some time soon for a Windows powershell version,

(new-object
Net.WebClient).DownloadString("https://builds.dawg.eu/install.ps1") | iex

though I'd prefer if someone else would be up for that task (I never use
Windows myself).

[¹]: https://github.com/MartinNowak/installer/commit/2d7c07066cfee1a610dda2026e4d4c4e7b41fa5f [²]: https://github.com/D-Programming-Language/installer/pull/162

I also intend to expand on that dmd-nightly semantic and add dmd-beta (using http://ftp.digitalmars.com/LATEST_BETA).

Once the nightlies are stable enough they should become a dlang.org subdomain, e.g. nighlies.dlang.org, and get a place on the download page. Together with the corresponding one-step installer script lines.

curl -fsSL https://dlang.org/install.sh | bash -s dmd
curl -fsSL https://dlang.org/install.sh | bash -s dmd-beta
curl -fsSL https://dlang.org/install.sh | bash -s dmd-nightly

## Progress on OSX shared libraries

I don't want to announce anything unfinished, but with Nicolas Jinchereau (bitwise) we finally seem to have found someone capable of putting together an OSX shared library implementation for dmd. Thanks for your great work so far.


##### Planning of next spring #####

I'll build 2.069.2-b1 tomorrow. The project tester [¹] didn't show any new issue and I'm not aware of any 2.069.x regressions being reported. Please test the beta asap, so we can ship 2.069.2 on time (2015-12-03).

Besides the nightlies, I only plan to finish the GC spinlock implementation [²] and ignoring GC.free in finalizers [³], as I'll be on vacation from 5th till 19th of december.

Maybe there is some extra time to track down the remaining slist_reset bug for OMF [⁴] (help appreciated) or finish the smart_ref library, but it's not very likely.


[¹]: https://github.com/MartinNowak/project_tester
[²]: https://trello.com/c/Xp44NwHG/87-ttas-spinlock-for-gc
[³]:
https://trello.com/c/1UXjGQBf/135-issue-15353-std-experimental-allocator-cannot-free-memory-in-its-destructor-if-the-gc-is-an-ancestor
[⁴]: https://github.com/D-Programming-Language/dmd/pull/5109



November 28, 2015
On 28 November 2015 at 04:58, Martin Nowak via dmd-internals < dmd-internals@puremagic.com> wrote:

>
> ## nightlies and installer scripts
>
> I spend a lot of time to setup nightly builds. First I was using my
> local dev machine with a rtc wakeup and cron job, but that turned out to
> be too unreliable (kernel/package updates, S3 wakeup hang, forgot to
> unlock gpg key). So I now migrated this to a dedicated server (a low end
> box that's 4x slower than my dev machine) and hope to stabilize and
> publish everything over the next few days.
> You can find a beta of the nightlies service here.
>
> https://builds.dawg.eu/
>
> If you want to install a nightly you might try out our new install script (and yes that safe to pipe to bash [¹]).
>
> curl -fsSL https://builds.dawg.eu/install.sh | bash -s dmd-nightly
>
> If you don't like that, do something else, e.g.
>
> curl -fsSLO https://builds.dawg.eu/install.sh
> curl -fsSLO https://builds.dawg.eu/install.sh.sig
> gpg --verify install.sh.sig
> vim install.sh
> chmod +x install.sh
> ./install.sh dmd-nightly
>
> or just download the latest nightly directly.
>
> https://builds.dawg.eu/dmd-nightly/dmd.master.linux.tar.xz
>
> The installer script with create a ~/dlang folder, installs itself, can
> install almost any version of dmd/gdc/ldc, is based on the travis-ci and
> heroku script and should be pretty self-explanatory.
> If you find something non-obvious, please help to improve it [²].
>
> I hope to find some time soon for a Windows powershell version,
>
>
If you can get one for FreeBSD, that would save me having to occasionally fix the nightly build script someone else wrote for asm.dlang.org (as of writing, it's sitting at DMD64 D Compiler v2.068-devel-0f3850d).


November 28, 2015
On 11/28/2015 10:45 AM, Iain Buclaw wrote:
> If you can get one for FreeBSD, that would save me having to occasionally fix the nightly build script someone else wrote for asm.dlang.org (as of writing, it's sitting at DMD64 D Compiler v2.068-devel-0f3850d).
> 
I already works on any UNIX platform we provide binaries for.
The only dependencies are bash, curl, xz, egrep, and optionally gpg.
Tools are required lazily, e.g. egrep is only needed for the list command.
Not all binaries are available for all platforms, e.g. there is no dub,
ldc, or gdc for FreeBSD.
https://github.com/MartinNowak/installer/blob/fe56deec0f8a7e51111f673e7acb5f71732a923b/script/install.sh#L532
https://github.com/MartinNowak/installer/blob/fe56deec0f8a7e51111f673e7acb5f71732a923b/script/install.sh#L356