On Tuesday, 5 January 2016 at 18:09:57 UTC, Andrei Alexandrescu wrote:

> Again this goes back to Adam. Let's say we had a contributor Eve who'd gladly take emailed questions and suggestions and integrate them. Would that be as nice?

Yes. But I would also be happy with something simpler. If there was a person not named Andrei or Walter in charge of the docs, and I could email them to propose a change, and they'd say, "I don't think that is an improvement so don't bother" or "Okay, but you need to do this for it to be accepted" I would be willing to create the PR myself. I would even be willing to help Eve by creating PR's once she has decided what needs to be done when others email with suggestions.

On Tuesday, 5 January 2016 at 14:18:32 UTC, Andrei Alexandrescu wrote:
> There's been a recent discussion with Walter and Martin about using wildcards in makefiles (which would obviate the necessity of being explicit about files).

I actually do NOT use wildcards in most of my own makefiles for exactly these reasons, I tend to keep dead files around.

But I also list the file once and only once. There's no separate SRC, MANIFEST, DOCS, whatever. I don't even know what the difference is between each of those.

However, that's for building code which is separate from building documentation. Building code needs a bit more care because it has to actually compile. Docs don't, it can make a few extra html files without lots of worry.

This is one of ddoc's fundamental technical flaws, by the way, being integrated with the compiler. The compiler *needs* everything set up right to function properly. You can't compile code without the proper conditional compilation settings, for example. You can't compile a program with two main()'s.

But docs are different. You don't want a function not to appear just because it is under a version(cool_feature) block.


Notice I said "fundamental" rather than "fatal" because we have a workaround: the version(D_DDOC) thing. But that sucks. You now have to write the whole function prototype again to show up in the docs and if it happens to be different on the other platforms, well, you might as well just write the docs by hand. You've lost all benefits of auto-generation now.


Oh yeah, and the website also has a navbar that is manually maintained, and an index page which is often neglected, probably because people don't expect to have SIX different places to add a new file! (The makefile parts, these macros, and, of course, the git add to the filesystem.)

My nav bar is automatic. Click around std.socket to see what I did with it today: http://dpldocs.info/experimental-docs-2/std.socket.Socket.setOption.4.html


> to produce incomplete builds if some files are missing by mistake.

That would surely be caught by the compiler itself (module file not found) or the test runner. In the case of documentation, a bunch of broken links would be detected. For druntime, the Phobos build would fail, etc.



> Would you recommend switching to wildcards in our makefiles and assume people use git to keep their directories in good shape?

It'd probably work fine in git; worth an exploration. Especially for doc building which wants everything in /import, without a doubt. Phobos isn't as clear cut because it builds right from /src and has some internal files in there, but still, a wildcard that excludes the internals beats one that is wrong in the common case.

> * Each library version must be built with the same compiler version. The appropriate compiler version might be missing, so it needs to be downloaded, installed, and run on the side without affecting any existing installation.

And this is again because everything is so strongly coupled to dmd. I'm not even convinced Phobos ought to be this closely coupled (my libs outside Phobos are compatible with at least a year of previous versions, sometimes more, and my users find this very valuable for stability against the fear of regressions), but the documentation certainly shouldn't be.

With an external doc generator, you can get much broader version compatibility since it isn't responsible for actually compiling the entire language like dmd. It just needs to parse enough of it to get useful information to attach to comments, and understand just enough to attach some useful text.

For example, my doc generator (built on Brian Schott's libdparse btw) knows that "@safe" is worth pointing out to the user. But it doesn't need to know what it actually means. It doesn't run that level of semantic analysis and is thus isolated by changes in the safed spec. dmd doesn't have that luxury.


> My understanding is that your system does not do any of these things.

I ask you to consider the importance of compromising website quality in favor of those things from a strategic viewpoint.

If the pdf is really popular enough that its continuance is warranted (tbh, I can't understand why it would be, it is equivalent to or inferior to even the current website in almost every way), you should find a way to do it that doesn't hurt the main site.

I certainly won't be concerning myself with it. I do care about offline viewing, and have a simple solution: I can offer a zip/tarball of the website html files. They will just work without a network connection as well.

BTW: generating those other formats from html is also pretty easy and can be done after the fact.

> The script https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak stands at 498 lines. What ways there are to make it considerably simpler and smaller?

I've looked at this before and didn't even understand what it is trying to do, much less why it is doing that. I don't feel comfortable modifying code that I don't understand.

And frankly, I have zero interest in dealing with complicated build systems. My view is if they aren't nearly invisible, they have failed me.

And I deal with failing subordinates in a similar manner to Darth Vader:

https://www.youtube.com/watch?v=aV2DLkDPwM8&feature=youtu.be&t=33s

> Indeed the web forms sound like a great idea. Would you want to work on such for the mainline website as well? It's not a contest; the better everything is, the better off everybody is.

It's not the web form that's difficult though, it is getting the new file integrated with the build and actually running it. The existing site makes that exceedingly difficult.

> Nice. What could we do in ddoc and ddox to have a similar improvement?

I implemented it as a magic macro, they could do the same.

This kind of expansion cannot be implemented as a ddoc macro itself because it needs more info than text expansion (and mine has an if statement if name lookup fails), but it is fairly easy to express in D.

> How do you envision scaling it up as the system becomes more successful and you get more requests than you can handle reasonably quick?

I'll cross that bridge when I get to it. Let's not let tomorrow's potential hurdle be today's showstopper.


> Would you be willing to set up such a system for dlang.org as well? It would be absolutely fantastic.

People can (and often do) email me all kinds of things. But I'm not the god-emperor of dlang.org so what I can easily do there is limited.

> Anyone could do the same today for all pull requests - just fetch them, fix them, and resubmit with credit. Additionally, team members could just pull the PR, fix it, and make an additional commit.

Indeed, that's not a technical barrier. But it is a cultural barrier - the D PR process (which btw is strongly encouraged by the GitHub UI, it isn't just us) has more emphasis on leaving comments than on fixing it yourself.

> I'll note that you are already a member of that team so you already had the rights to exact quite a few of the improvements you intend to make going forward. Would you please join me in taking a look at older PRs (oldest is from 2014) and pull them in?

The technical access and the feeling of ownership are two different things.

If you gave me a key to your house, would it be fine for me to do in some day when you're not home and start remodeling it?

That's the problem with the "lieutenant" thing that's been discussed before. Lieutenants are junior officers who just keep things running according to regulation while the captain is busy elsewhere. If anything extraordinary happens, they report it up the chain of command.

What we need is a greater amount of delegation. Lieutenant-generals perhaps: individuals who have full authorization and broad autonomy to run an entire sector of operations as they see fit.

> Overall my understanding of your message is "My system would be better not for a fundamental technical reason, but because I am willing to pour in it time and talent." This is the kind of argument I have a lot of respect for.

Well, there are technical reasons too. Tying ddoc to dmd was a short-term win, it got to reuse the compiler to do hard work at a time when there were no realistic alternatives.

But, today, it is limiting. Sharing code with the compiler means ddoc is subject to the compiler's trade-offs.... which, given a choice that strengthens code or strengths docs, code is going to win every time. As it should, that's the compiler's job. Moving ddoc to a separate path will result in code duplication at best, so at that point, you don't gain much from staying in dmd anymore.

Ddoc's text macro system, while an elegant implementation and good enough for many things, is also a fundamentally limiting factor (and now tied up a bit due to backward-compatibility with arbitrary user-defined extensions), and prone to complications. It puts a limit on how much analysis of the source text we can realistically do because anything could expand to anything else. It is a  mistake. (The missing word there btw is $1, disappeared because ddoc mistakes it for syntax and just erases it. Seriously, try it.)

Ddoc's total failure to understand what it is generating means it cannot automatically link. And since it lacks conditionals, you can't even program it to link to user-defined things. (And no, I am not suggesting we add $(IF). We don't need yet another programming language... nobody wants to write their documentation in uncommon Lisp. We want it to just work.)


Fixing these will mean breaking backward compatibility. If you're willing to do that, great! I've been writing a ddoc 2.0 generator :P

On Tuesday, 5 January 2016 at 14:18:32 UTC, Andrei Alexandrescu wrote:
> Regarding PRs that are not looked at, we currently have 18 PRs at https://github.com/D-Programming-Language/dlang.org/pulls, and 16 folks who have the rights to pull them (https://github.com/orgs/D-Programming-Language/teams/team-dlang-org). I'll take a look at them after this (some look really trivial); it's a bummer they aren't looked at more regularly.

I'm usually subscribed to the dlang.org GitHub repo and merge PRs or work with PR contributors into getting their PR into merge-able state. However, I'm currently abroad so not very responsive. PRs which affect the language specification are labeled as such and need to be reviewed/merged by someone more familiar with the subject, though.

I'll throw another idea into the discussion based on the potatoes from my side of the table. Digger can already build the website documentation with one command, albeit only on Posix. How would it sound to expand that to Windows, and to make it easier to contribute to D via Digger without having to touch Git (e.g. using a new "digger pull-request" command)? This would only make sense if Digger was promoted as a primary means of building D, rather than one of the alternatives / last-resorts as it is currently.

On Tuesday, 5 January 2016 at 18:09:57 UTC, Andrei Alexandrescu wrote:
> On 01/05/2016 11:35 AM, default0 wrote:
>>
>> Yes, and because its lots of effort flowing into something that D is
>> usually very fond of: Simplicity.
>
> I'll be curious how the simplicity theme keeps when needed features get added. dlang.org started very simple and grew by accretion.

From Adam's responses it would seem he is going to ignore most of the features the original dlang.org has in favor of focussing on HTML documentation.

>> I remember looking at a part of the documentation and wanting to suggest
>> a change of wording in a few sentences, since I felt like they didn't
>> convey information clear enough, but after seeing that I would have to
>> somehow best-guess my way through tons of macro syntax without clear
>> indication on where to look for explanations of anything I bailed. Not
>> about to spend half a day or a day to suggest changing three sentences
>> and be told that they were fine as is.
>
> Is the recent http://wiki.dlang.org/Contributing_to_dlang.org along the lines of what you need? What other sort of documentation would you find useful?

Yes, this is helpful in providing background on how to make changes. That said, the amount of content on this page is telling that it'll be an involved process to set stuff up, especially if at some point in my setup I get errors that are neither explained nor straightforward to track down. So while this is helpful, it also makes it clear that there's a not-low barrier to entry if I do want to introduce a change - which is fine, except if it's documentation that barrier to entry should be actually formulating coherent, understandable and thorough documentation rather than a build system.

>> Sending Adam an E-Mail is
>> something I totally would've done though :-)
>
> Again this goes back to Adam. Let's say we had a contributor Eve who'd gladly take emailed questions and suggestions and integrate them. Would that be as nice?
>
>
> Andrei

Much nicer than reading a somewhat long page on how to set up a new development environment and fiddling with it until it does what I want, followed by trying to guess what macros I need to invoke for displaying what things, followed by checking if it renders as expected or if I accidentally used a "," incorrectly somewhere instead of just formulating a documentation text that is hopefully an improvement over what existed prior to it (or filling a void that used to be there) :-)

In the end most of this comes down to a lack of motivation: I'm fine trying to improve documentation text if I see an issue about it, but if that entails stopping what I was originally doing for so long that I will possibly forget what I was originally doing (ie requires me to read documentation on how to set up a whole new development environment), I will usually decide that nah, it isn't THAT important.

While I could obviously reuse the setup after I did it once, I'm not keen on doing it once, I do happen to be a really lazy person about setting things up (not so much doing actual work on something that is already set up, though) because I tend to get really weird errors for really weird reasons when trying to (not in particular with D - I just always seem to be doing everything wrong when setting anything up).

On Monday, 28 December 2015 at 20:15:30 UTC, Adam D. Ruppe wrote:
> http://dpldocs.info/findSkip

We already have a nice and powerful documentation generator called ddox.
https://github.com/rejectedsoftware/ddox
It also contains an experimental libdparse based input (rather than relying on dmd's json ouput).
It is flexible enough to be templated/styled very differently, see https://dlang.org/library/index.html, https://vibed.org/api/, https://github.com/MartinNowak/scod (martinnowak.github.io/bloom/bloom.html).
Its behavior can be customized, the project is actively maintained, ddox compatible tools can easily be integrated with dub, and whatever you want to change for template constraints should be easy to achieve.

On 2016-01-06 05:29, Adam D. Ruppe wrote:

> I actually do NOT use wildcards in most of my own makefiles for exactly
> these reasons, I tend to keep dead files around.

"git ls-files" should take care of that.

-- 
/Jacob Carlborg

On 2016-01-05 15:18, Andrei Alexandrescu wrote:

> There's been a recent discussion with Walter and Martin about using
> wildcards in makefiles (which would obviate the necessity of being
> explicit about files).
>
> My understanding is that build scripts (including makefiles) are
> recommended to include a manifest list of files that are part of the
> project, as opposed to picking up files with wildcards. That way there's
> less likelihood to pick up litter along with the legitimate files, or to
> produce incomplete builds if some files are missing by mistake. The
> underlying thesis is that that's a good kind of redundancy.

druntime already uses wildcards [1], if I read the makefile correctly, in some places.

[1] https://github.com/D-Programming-Language/druntime/blob/master/posix.mak#L147-L159

-- 
/Jacob Carlborg

On Wednesday, 6 January 2016 at 10:25:50 UTC, Martin Nowak wrote:
> We already have a nice and powerful documentation generator called ddox.

You say that like I've never hard of it before, when I've spent quite a few words over the last ten days writing up my critiques of it, including both bugs and questioning its fundamental approach.

I guess I'll write more about it. I really wish you guys would actually read the arguments you're replying to though before posting. This is very frustrating and contrary to popular belief, I'm not a saint with infinite patience.

> https://github.com/rejectedsoftware/ddox
> It also contains an experimental libdparse based input (rather than relying on dmd's json ouput).

I confess I didn't actually know it had that (it has zero mention... anywhere outside the source)... though the fact that it is there actually strengthens my position! Look at the ddox issues list, marked external. Those are from dmd... but they aren't all bugs. The JSON isn't just for docs, and when there's a trade off between doc and whatever else people use the json for, the doc side often loses.

e.g.:
https://github.com/rejectedsoftware/ddox/issues/19

Apparently, the ddox team agrees with me that being wed to dmd is a flawed approach.


Though, looking at its source, it doesn't actually use libdparse for much beyond the basics - just getting enough to feed into the existing core, which reflects its preference for the inferior json, and it appears to be stalled there. It is still limited by its initial decision of going down the dmd route.

> It is flexible enough to be templated/styled very differently

That's trivial. Even the worst html file is flexible enough to be templated/styled very differently because that's part of html's core nature.

> the project is actively maintained

With about 1/3 of its github bugs still open, including about half its open bugs more than one year old. About half its D bugzilla bugs are still open, including ones over 2 1/2 years old.

I know projects get bugs open when they are used, but ddox is a one-person project and that one person doesn't seem terribly active in it.

https://github.com/rejectedsoftware/ddox/graphs/contributors


Looking at the bug list, lol:
https://issues.dlang.org/show_bug.cgi?id=14608

It was open for six months without so much as a comment. The thing Andrei is asking for and Sonke agrees is a good model... is exactly the way I did mine the first time.

http://dpldocs.info/experimental-docs-2/std.algorithm.searching.OpenRight.html



And yes, yes, I'm sure if I spent 60 hours on ddox over this Christmas instead of on my new system, some of those bugs would be closed. But I betcha I would have hit some annoying wall and instead spent the time on my paying job or something (I do have a big tax bill looming and probably should be doing paying work!), so that's all hypothetical.

And when I did open a pull request, it'd probably sit there for over two months without comment like the other ones that are open right now. I'd have to fork the site anyway to get any changes live!

> ddox compatible tools can easily be integrated with dub, and whatever you want to change for template constraints should be easy to achieve.

The template constraint formatting is just the beginning.

On Tuesday, 5 January 2016 at 18:09:57 UTC, Andrei Alexandrescu wrote:
> Again this goes back to Adam.

It occurs to me, looking at the status quo, that a single point of failure is more robust than several points of failure in series.

/library/ depends on the dlang.org, Phobos, and ddox.

dlang.org depends on.... nobody really knows who owns it, but it seems to be you.

Phobos depends on a few different people, making it the most reliable in the group, but it is still practically coupled to dmd.

ddox depends on Sonke and dmd (practically).

dmd depends on Walter.


Any any person in that component drops the ball, the /library/ code gets stalled. The fix might get into ddox but not on the website. It might depend on something in Phobos. It might need dmd's output to change.


With my system, it is all under my control. As supreme god-emperor of my domain, my server, my phobos fork, my generator, my parser fork, and my work process, yes, you'd be in a bit of trouble if I went away (though pretty soon you'll just be able to fork it yourselves in that event).

But at least I'm never blocked by someone else's problems.


> Let's say we had a contributor Eve who'd gladly take emailed questions and suggestions and integrate them. Would that be as nice?

Poor Eve, on the other hand, would be often blocked by somebody else outside her control.

On 01/06/2016 07:27 AM, Jacob Carlborg wrote:
> On 2016-01-06 05:29, Adam D. Ruppe wrote:
>
>> I actually do NOT use wildcards in most of my own makefiles for exactly
>> these reasons, I tend to keep dead files around.
>
> "git ls-files" should take care of that.

Thanks, Jacob I think this is a great idea. If anyone would like to initiate replacing some uses of file lists with git ls-files, please go ahead. And it should work on Windows, too.

The remaining issue is that that makes the makefile assume git is installed. Is that reasonable?


Andrei