I just watched Andrei's *Quo Vadis *talk from this year's DConf and I liked it a lot. I strongly agreed with his notion that the D community needs to embrace "operational professionalism"; I've been thinking along those same lines for months and it's the one thing that the D community really, *really * needs to work on. I've been a huge fan of the D language ever since I read TDPL ~a year ago. IMO it's the most amazing language I've ever used, and I've used far too many.

I don't normally participate on the D mailing lists (I lurk often though), but my conscience was gnawing at me to send this email. It's a bit long, so bear with me.

So, Andrei also mentioned how the community needs to improve the
dlang.orgsite and make it a first-class priority. It's the point of
contact for
newcomers so it's very important we get it right. Since I know a
thing-or-two about web development, I decided to look into improving the
site.

There's some low-hanging fruit with the site that shouldn't be too hard to fix: there's an incredibly annoying "content flash" after any page is loaded (it loads fine, you see the text, then the text disappears for a second, then appears again). I've cursed at this more than a dozen times; it's especially annoying on any mobile browser.

For a site that's just static content it also takes a long time to load.
Inexcusably long TBH. The PageSpeed analyzer puts the load time at ~7
seconds for a big page from the library
reference<https://developers.google.com/speed/pagespeed/insights#url=http_3A_2F_2Fdlang.org_2Fphobos_2Fstd__datetime.html&mobile=false&rule=____critical__path>.
In my personal experience there's no good reason why any 100% static site
should take more than 1s to load on a fast desktop machine, and often you
can get it to 0.5s with a bit more work. If it's loading in more than 1s,
then something is being done in a sub-optimal way. But that's not
surprising because building a site so that it loads optimally takes a fair
bit of doing and a large amount of domain knowledge.

So I decided to work on it a bit. Here's a blow-by-blow description of my attempts to do so. All of this happens over the course of a few hours.

   1. Run the site through the PageSpeed analyzer and realize there's lots
   of low-hanging fruit. I can fix all of those and make the site faster,
   great.
   2. Looking at the Network data in the Chrome DevTools, realize the site
   isn't using a CDN. Make a note of mentioning this to the site maintainers;
   using a CDN is critical to performance and ever since CloudFlare came on
   the scene (fast & free CDN that's also trivial to set up) there's no excuse
   not to use one.
   3. Record a Timeline profile of the site's loading process in the Chrome
   DevTools. Study the profile and realize there's quite a bit extra reflow
   and repaint going on. Those should be addressed as well.
   4. Record a JavaScript profile of the site with the DevTools JS profiler
   and the new Flame view. With this information and the Timeline profile,
   realize that hypenate.js is probably the culprit of the content flash bug.
   Still have to verify this, it might be something else.
   5. Notice the site's homepage is throwing JS exceptions; that will need
   to be fixed as well. OK, time to checkout the site repo and make changes.
   6. Go to the D-Programming-Language/dlang.org github repo page.
   7. Wait, where's the README? No README? What? How do I make heads or
   tails of this thing?
   8. Oh God, the site is written in ddoc. Dear God *why*? Whatever, I'm
   not trying to change the content, just the framework that delivers the
   content. Complaining about the choice of markup language used for the
   content is not a fight worth fighting and the discussion would probably not
   be productive.
   9. Spend X minutes trying to figure out how to build the site. Find
   "wind32.mak". Sigh. I use Mac OS X and Linux.
   10. Search for "unix", "linux" and "makefile" in Chrome on the repo
   site. Nothing useful. Is there no way to build this site on a unix machine?
   11. Notice there are several outstanding pull requests. Look through
   them to figure out if there's any useful information there.
   12. Through nothing but sheer luck, notice a pull request that makes
   changes to "posix.mak". Ah, so there's an important file I missed. An ~hour
   down the drain because there was no README and I very easily could have not
   even found the relevant file.
   13. It appears that the makefile assumes the repo checkout needs to be
   performed inside of a specific folder structure that has other repositories
   already checked out. Great, so the build is not hermetic. Is continuing to
   try to make this work going to be worth my time? Let's look at how the
   project handles pull requests.
   14. Notice that there are several pull requests that have been waiting
   for a response for weeks and months. So even if I figure everything out
   (probably not too complicated from this point forward) and successfully
   make the relevant changes, my pull requests would probably languish
   unattended for a very long time.
   15. Conclude that this is not worth my time. "Meh, someone else will fix
   the site."

That last point was several hours ago. Since then my conscience has forced me to at least point out the issues I encountered so that project leaders might do something about it so that possible future contributors don't have to go through the same painful process.

Criticism that's not constructive is not useful. So here's several actionable items for the project leaders:

   - Prompt response to pull requests from outsiders is of the utmost
   importance. Response latency of anything more than a few days sends the
   signal that the community doesn't care about external contributions. To
   paraphrase Andrei, "if you want to be an OSS project with many
   contributors, do what projects with many contributors do". This entails
   promptly responding to external developers. Huge and successful projects
   like Ruby on Rails and Twitter Bootstrap can afford to have a high response
   latency because they have countless thousands of people trying to
   contribute to them. Causing those contributors some pain is not going to
   materially impact the success of the project (IMO of course they still
   shouldn't do this, but they can afford to). D is not there yet and should
   go out of its way to facilitate ease of contribution.
   - Good, up-to-date developer documentation is necessary if you want
   others to contribute. People can't make changes if they can't easily build
   the project. Sure, determined people could figure it out after a while even
   without the docs, but these roadblocks on the road of success don't help
   the project and are discouraging to outsiders. All of the
   "D-Programming-Language" repos on GitHub should at the very least have a
   README file that points newcomers in the right direction and answers the
   most basic questions they might have.
   - The issue tracker for the dlang.org repo has been turned off. I now
   realize that this is probably because the D community uses their bugzilla
   tracker and wants issues reported there, but nothing on the
dlang.orgrepo page says this. There are no docs that say "hey use this
bugzilla
   tracker to report issues". Nothing on the dmd repo page says this either.

I hope I have been constructive with my suggestions and that some of them will be acted upon.

Wow, this ended up being far too long.

Would it be possible to add links to national dlang communities to the bottom of the page to make comers easier to find information and help on their native language?

On 2013-07-14 05:19, Val Markovic wrote:
> I just watched Andrei's /Quo Vadis /talk from this year's DConf and I
> liked it a lot. I strongly agreed with his notion that the D community
> needs to embrace "operational professionalism"; I've been thinking along
> those same lines for months and it's the one thing that the D community
> really, /really/ needs to work on. I've been a huge fan of the D
> language ever since I read TDPL ~a year ago. IMO it's the most amazing
> language I've ever used, and I've used far too many.
>
> I don't normally participate on the D mailing lists (I lurk often
> though), but my conscience was gnawing at me to send this email. It's a
> bit long, so bear with me.
>
> [Snip]

I completely agree with you and I think it madness that the site is written in Ddoc. But apparently Walter and company loves it.

-- 
/Jacob Carlborg

Jacob Carlborg:

> I completely agree with you and I think it madness that the site is written in Ddoc. But apparently Walter and company loves it.

Do you like this? It's made with Python:
http://sphinx-doc.org/

Bye,
bearophile

On Sunday, 14 July 2013 at 04:56:46 UTC, Val Markovic wrote:
> ...

Thanks for the input!

I do personally agree that official web page is one of most lacking areas of D public image. Maybe that is because there are no people among core developers specializing on web development (or am I ignorant here?).

To be honest I still have no idea how this web site was created and what is the real process behind maintaining it. Guess there are two options - either it is made more transparent by current maintainers or someone steps up to take the responsibility to tidy things up (not me!).

Using DDOC itself does not seem that critical IMHO - it is just one more markup language which anyone working with D knows at least vaguely. But maintaining whole stuff via git repo + makefiles is definitely cumbersome, no idea why that was chosen.

On 2013-07-14 14:34, bearophile wrote:

> Do you like this? It's made with Python:
> http://sphinx-doc.org/

Ddoc in it self is not that bad, for documenting code. It's just that I would never create a web site using it. I would go with a server side script. I prefer Ruby on Rails for that.

-- 
/Jacob Carlborg

On 7/14/2013 2:30 AM, Jacob Carlborg wrote:
> I think it madness that the site is written in Ddoc.

Different, sure. Madness - why?

On 7/14/2013 6:49 AM, Dicebot wrote:
> But maintaining
> whole stuff via git repo + makefiles is definitely cumbersome, no idea why that
> was chosen.

1. github is how the entire rest of D is maintained, it would be strange if the website was handled via some completely different system.

2. github is of proven utility in enabling anyone to contribute to the web site via pull requests.

3. github/makefiles are how everything else in dlang is done, so contributors should already be familiar with the process.


As for using Ddoc,

1. It relies on D being installed. Presumably, anyone interested in developing the D website already has D installed. No other tools are necessary.

2. Ye Olde Dogfooding argument. Experience with the web site has driven some good improvements to Ddoc.

3. HTML, PDF, Ebook, and CHM outputs are generated from Ddoc.

4. Website can be built on any platform that supports D.

This discussion seems to have gone off track.

I pointed out three things that I thought should be addressed to ease the use of contributing to the dlang.org site (and other D-project components), and those three things were: a) lower response latency on pull requests, b) better documentation of the component "plumbings", specifically how to build the dlang.org website and c) README files in all the D-project repos with answers to common questions outside developers might have, like "where do I report bugs?".

The discussion here goes on to debate the pros and cons of ddoc which is * explicitly* something I mentioned as "not worth fighting over". Yes, IMO it's an unfortunate choice of a markup language because to outsiders it's esoteric and unlike most other markup languages and there are far better options out there (Markdown or reStructuredText). For a project that's still looking for contributors it's just another roadblock to adoption. It's what they call an "unforced error" in tennis; something that makes it harder to succeed for no benefit or reason, and not doing it (or doing it differently) helps you get to where you want to go faster.

But again, this is beating a dead horse. This mailing list has seen plenty of discussions on the topic and the powers that be haven't been convinced to move away from ddoc so I don't think it's going to be fruitful to debate this further. Let's be practical and let it be and move onto other, more important problems. Nobody has ever said "Oh that D language is great but ddoc is so bad I've decided to use Java". That just doesn't happen. It's not that important.

Ddoc was irrelevant to the changes I wanted to make to the site anyway. The more important problems I listed were the ones preventing me from making them.

Other points:

1. Having the website inside git on github was a great idea. Walter made all the right arguments explaining why.

2. Makefiles are fine, but there needs to be a README saying "hey go run that posix.mak file" because there's no "Makefile" file in the site repo. And then the makefile has assumptions about its environment which are not documented anywhere. It's not obvious what needs to be done.

3. Ruby on Rails (or Django or any other web framework that dynamically builds a server response) would be a terrible choice for a static website. It would be nothing but overhead for no benefit. Nothing on dlang.org is dynamically generated, nothing talks to a database (nor should it). The site can and should be generated from some simple markup language. The end result should be HTML, CSS, JS and image files that are then just served by the simplest of servers. For these types of sites, the industry standard is quickly becoming to serve the files directly out of Amazon S3 with a CDN (like CloudFlare) in front of it. The end result is an incredibly fast browsing experience because everything is just served directly out of the CDN edge servers without any kind of dynamic response generation. Requests come in for files and the CDN just echoes them out of memory to the network.

Of course, forum.dlang.org would be different, but (internally) that's a
different site entirely.


On Sun, Jul 14, 2013 at 1:35 PM, Walter Bright <newshound2@digitalmars.com>wrote:

> On 7/14/2013 6:49 AM, Dicebot wrote:
>
>> But maintaining
>> whole stuff via git repo + makefiles is definitely cumbersome, no idea
>> why that
>> was chosen.
>>
>
> 1. github is how the entire rest of D is maintained, it would be strange if the website was handled via some completely different system.
>
> 2. github is of proven utility in enabling anyone to contribute to the web site via pull requests.
>
> 3. github/makefiles are how everything else in dlang is done, so contributors should already be familiar with the process.
>
>
> As for using Ddoc,
>
> 1. It relies on D being installed. Presumably, anyone interested in developing the D website already has D installed. No other tools are necessary.
>
> 2. Ye Olde Dogfooding argument. Experience with the web site has driven some good improvements to Ddoc.
>
> 3. HTML, PDF, Ebook, and CHM outputs are generated from Ddoc.
>
> 4. Website can be built on any platform that supports D.
>
>

On 7/14/2013 2:18 PM, Val Markovic wrote:
> I pointed out three things that I thought should be addressed to ease the use of
> contributing to the dlang.org <http://dlang.org> site (and other D-project
> components), and those three things were: a) lower response latency on pull
> requests, b) better documentation of the component "plumbings", specifically how
> to build the dlang.org <http://dlang.org> website and c) README files in all the
> D-project repos with answers to common questions outside developers might have,
> like "where do I report bugs?".

I agree with your three points. I'll add that anyone can do a pull request to address any/all of these.


> The discussion here goes on to debate the pros and cons of ddoc which is
> /explicitly/ something I mentioned as "not worth fighting over".

Sure, but I wanted to address the comments that it was "madness" to and "no idea" why to use ddoc. Yes, it's come up before.