On Monday, 28 December 2015 at 23:05:28 UTC, Andrei Alexandrescu wrote:
> (a) is the new proposed system differentiated enough to justify its existence and motivate others to join in?


I was just watching my newbie friend try to manipulate directories in D. His first instinct was to go to std.path. I decided to edit std.path to add a note in the header "if you want to get path names from a directory, use std.file".

Linking to std.file proved to be a huge hassle. Sure, I could just $(LINK2 std_file.html, std.file) like Phobos does in other places, but alas, that breaks ddox.

http://dlang.org/library/std/algorithm.html

Click the "See also Reference on ranges" there and observe the 404.

So I wanted to define a new macro called MREF so you'd write it $(MREF std,file) and it would replace with underscores on ddoc, slashes on ddox (and dots on my docs).

Easy, right? Wait, XREF doesn't work right on modules, it expects the second arg to be a function and links break in some places if you leave it blank, and it doesn't work on submodules... can't there just be a REF thing?

I guess not, the system is too complicated. Whatever though, I'll make my new MREF, or module reference. (Am I seriously the first one to do this?)

	Check out $(MREF std,file) and $(MREF std,algorithm,searching)!

	Macros:
		MREF_HELPER=_$1$(MREF_HELPER $+)
		MREF=$1$(MREF_HELPER $+).html

Wow, it works! (Well, basically. It doesn't work if the module has only one name, like object.d, but we can special case that one.)


But now, where do I put it? I grep for XREF in dlang.org/*.ddoc and find a few candidates then check the wiki http://wiki.dlang.org/Contributing_to_dlang.org#Macro_Definition_Batteries:_.ddoc_files to see what it says.

...it returned files that aren't listed there... what should I do with them? Oh dear this is taking too much brain power, I just wanted to add a sentence linking people to the other module!




BTW this is why my thing separates all the filenames with dots, just like D itself does. Predictable, even without semantic analysis!


This huge friction has killed my desire to contribute to Phobos before and it looks like it is again.

the difference is this time, I have my own fork so the community doesn't have to lose out.

On Wednesday, 30 December 2015 at 23:05:11 UTC, Adam D. Ruppe wrote:
> On Monday, 28 December 2015 at 23:05:28 UTC, Andrei Alexandrescu wrote:
>> (a) is the new proposed system differentiated enough to justify its existence and motivate others to join in?
>
>
> I was just watching my newbie friend try to manipulate directories in D. His first instinct was to go to std.path. I decided to edit std.path to add a note in the header "if you want to get path names from a directory, use std.file".
>
> Linking to std.file proved to be a huge hassle. Sure, I could just $(LINK2 std_file.html, std.file) like Phobos does in other places, but alas, that breaks ddox.
>
> http://dlang.org/library/std/algorithm.html
>
> Click the "See also Reference on ranges" there and observe the 404.
>
> So I wanted to define a new macro called MREF so you'd write it $(MREF std,file) and it would replace with underscores on ddoc, slashes on ddox (and dots on my docs).
>
> Easy, right? Wait, XREF doesn't work right on modules, it expects the second arg to be a function and links break in some places if you leave it blank, and it doesn't work on submodules... can't there just be a REF thing?
>
> I guess not, the system is too complicated. Whatever though, I'll make my new MREF, or module reference. (Am I seriously the first one to do this?)
>
> 	Check out $(MREF std,file) and $(MREF std,algorithm,searching)!
>
> 	Macros:
> 		MREF_HELPER=_$1$(MREF_HELPER $+)
> 		MREF=$1$(MREF_HELPER $+).html
>
> Wow, it works! (Well, basically. It doesn't work if the module has only one name, like object.d, but we can special case that one.)
>
>
> But now, where do I put it? I grep for XREF in dlang.org/*.ddoc and find a few candidates then check the wiki http://wiki.dlang.org/Contributing_to_dlang.org#Macro_Definition_Batteries:_.ddoc_files to see what it says.
>
> ...it returned files that aren't listed there... what should I do with them? Oh dear this is taking too much brain power, I just wanted to add a sentence linking people to the other module!
>
>
>
>
> BTW this is why my thing separates all the filenames with dots, just like D itself does. Predictable, even without semantic analysis!
>
>
> This huge friction has killed my desire to contribute to Phobos before and it looks like it is again.
>
> the difference is this time, I have my own fork so the community doesn't have to lose out.

Thanks for doing this Adam. The official docs are a mess. Not just what you see when you look at the website, though there are important problems with that. It's the process that requires so much overhead that nobody wants to contribute. I really tried to do so myself, but I'm busy, and it is senseless that 95% (or more) of the time I spend on it is wasted due to a system that is flawed from top to bottom. The only thing that surprises me is that there are any contributions.

If you can make it easy to contribute, that is reason enough to push forward, even without the other improvements you are offering. The official docs can die a slow death and eventually Google will send newbies to your site.

The only request I make is that you continue to do this yourself in order to keep it out of overhead hell. I'll be using it for sure.

On Thursday, 31 December 2015 at 00:04:03 UTC, bachmeier wrote:
> It's the process that requires so much overhead that nobody
> wants to contribute. I really tried to do so myself, but I'm busy, and it is senseless that 95% (or more) of the time I spend on it is wasted due to a system that is flawed from top to bottom. The only thing that surprises me is that there are any contributions.

Exactly. I've put my time into ddoc. Let me give you a brief history of my efforts:

~2010: I had just written this awesome dom.d library and wanted to document it and release it to the world. I write stuff like:

/// Returns the text in the element. For example, innerText of <span>foo<span> is "foo" (without quotes)
string innerText();

And it came out mangled because of ddocs embedded HTML "feature" which doesn't encode the output. I proposed a fix, using ddoc's existing ESCAPES macro, and wrote the patch for dmd. Embedded html would be automatically encoded, but you can still define it in macro definitions, which can be inline with the comment, so it isn't hard to use when you want it.

It was rejected. Walter didn't see what the problem was and I was told to just write $(LT)span$(GT)foo$(LT)/span$(GT). Seriously.


Well, that was unacceptable so I decided to take matters into my own hands. On Feb 20, 2010, I registered dpldocs.info and started writing my own web service to delver dom.d's docs, encoding the output correctly, getting the docs out of dmd's JSON output, using -D -X.

Well, it worked, but it sucked. The JSON output didn't give me enough information to do cross referencing, so I wrote a little search engine to paper over it... and that didn't help much.

I moved on to other things, every so often still advocating to fix ddoc's escaping, but mostly just writing my docs such that they were readable in the source code and not bothering with generating them at all. (I'm told my source tends to be fairly readable anyway and I like to narrate my thoughts a bit in the comments including of implementation details so they understand the pros and cons of my decisions.)

At some point around here, I wrote a crappy Javascript table of contents for the D website (then hosted at digitalmars.com still) - the infamous "Jump list". This was meant to be a temporary measure until ddoc got proper table of contents support.

That sucky Javascript hack is still live today.


2011: We had a proposal for a website redesign. Remember this one? http://arsdnet.net/d-web-site/index.html

The jump list hack garnered a lot of hate. It was a blob of text that we couldn't read, but ddoc's design made a user-defined table of contents extremely difficult to implement. A few attempts were started, but none finished.

I wrote a program, called improveddoc.d, still on the 'net: http://arsdnet.net/d-web-site/improveddoc.d that post-processed the generated HTML to create a cheat-sheet table.

This is what it made of std.algorithm at the time:

http://arsdnet.net/d-web-site/std_algorithm.html


I offered to integrate it into the website tools... and got my first look at the horror that is that build process. Nevertheless, running a little post process script to run that on the html was doable. It also would read tag macros and use them to organize the content.

The idea (and working program) was rejected because the team felt a post-processor was the wrong way to do it.

Instead, Andrei manually wrote a std.algorithm cheat sheet and we added a bullet between the links in the jump list hack. Both of these have remained basically unchanged to this day and dmd still cannot generate a table of contents in ddoc.


We did get a website redesign after that, d-programming-language.org was launched, and Vladimir's forum got integrated (I wrote one too but his was so much better than mine that I gladly yielded). ddoc, however, barely changed.


For the next couple years, despite ddoc being unusable for half my libraries because they are web libs and showing HTML examples was practically impossible, I still was basically on its side, while still working on improving the json support. I had tried the json thing and found the compiler approach to be better, but the json output is good for a few things and Ddoc's syntax isn't all that bad if you aim for source readability and clever use of recursive macros can do some pretty cool things.

I put my support behind dmd. With the vibe.d team also pushing for improved json support, a few things happened there... though little of consequence. My opinion of the json output hasn't significantly changed since 2010: cool for a few things, I like that we have it, but it is not a substitute for compiler-integrated ddoc.


We got another website redesign in these years, and moved to dlang.org. The build process finally got documented after I and others complained for a while, but it was never actually cleaned up.


At the end of 2014, yea, even a year before this writing, people were complaining about ddoc's syntax again and asking for some Markdown variant to be introduced. Seeing an opportunity to revive my old encoding problem and finally write some ddoc for myself, I jumped on the `code` syntax and implemented it, including proper encoding of the output. Finally, I can write `<span>foo</span>` and have it legible to the user!

And with it also alleviating the pressure for syntax sugar from the community... this patch was actually accepted this time! For the first time in five years, I was excited about ddoc again.

I started using it to write This Week in D (and noticing the limits the macro syntax place on legible prose source...) and wrote at some length about my simpledisplay.d, cgi.d, and others (though not really much on dom.d, ironically. I still haven't gotten around to updating that.)

Just a few months ago, I got the simpledisplay docs looking pretty good with ddoc: http://arsdnet.net/arsd/simpledisplay.html


2015 was a good year for ddoc and I, we basically got along, despite its warts.

Then, October rolls around. A friend of a friend is looking for work and I say maybe you can help me with some D stuff. Don't know D? Not a problem, it is easy!


It wasn't easy. From installation, problems arose. I'd then show how to fix it and explain why.

She'd say "I understand now, but why don't the docs just say what you just said?"


Good question. I decide to start editing it in... and quickly hit a brick wall that kills my momentum. I couldn't get through the nonsense before having to get back to work.

I handle hundreds of D support requests in a year. Sometimes, I spend so much time on the chat, the learn forum, or Stack Overflow that I feel like I'm a full-time D support engineer. Time and time again, people ask things that ought to be pretty easy to find, yet they ask us anyway.

Maybe they just don't like searching and ask first. I actually don't mind that - I tell my coworkers that if their attempts at Google don't yield results in one minute, send me an IM or email (and continue looking). There's a good chance I'll have an answer in my brain somewhere, or at least a helpful pointer, and I don't mind the interruptions at all (if I didn't want them, I'd just close the chat client!). Better to ping me than waste hours on a fruitless search.

But still, some people say "I tried searching and couldn't find it"... and I totally believe that with D. I used it think it was just because we were young, but it has now been almost SIX YEARS since my first dive into the website issues and it has barely changed. There must be something systemic.


Fast-forward to December. We have that base64 question. I go to edit the site and get stopped by the process again. Nevertheless, I blamed the makefile and saw new documentation Andrei wrote on this. Maybe there's still hope.


Then another friend of mine tells me he finally started looking at the D Cookbook I gave him last year and is writing some code. I watch him work through a couple problems... and notice that the website was only useful to him insofar as he could copy/paste examples. As soon as he modified them and got an error message or tried to put functions together without pre-written examples, he'd slow way down. He'd get through it but not easily.

The D leaders know how important examples are. We are often told adding more is low hanging fruit. I completely agree. But that's not ALL we need. He wants examples to get started, yes, but he also wants understanding to go beyond examples. That's where text helps. That's where understanding the function signatures help. Eventually, when people go to write their own libraries, they might want to do Phobos style genericity. You can't do that without understanding the function signatures... and those blobs of text are not understandable.

Shift to the forum. We got talking about website redesign on the forum again a couple weeks ago and I remark how I think just whitespace formatting of the constraints would help a lot and almost went in dmd to implement it.

I just got sidetracked doing a quick paragraph fix for Andrei... and then wanted to see how far I could go. I got it working, despite the obstacles that ddoc's freeform macros bring, and surprise, it was rejected. Even the simpler patch that just collapses blank lines sits unanswered. A tool to look for broken links sits unanswered. There's always a call to contributions, but when you do push through the painful process to get the code up (and the tester, the f$^%$^ing tester), nobody seems to care.


The combined experience over these last six years tells me that the website sucks and it keeps sucking because changing it sucks even more than using it. A new strategy was required.

And here we are.




I still need to figure out how I want to add new docs and link them all in, but I guarantee you it won't consist of editing SRCS and MANIFEST and posix.mak and win32.mak and std.ddoc and latex.ddoc and <voice trails off>...

On 12/30/2015 06:05 PM, Adam D. Ruppe wrote:
> the difference is this time, I have my own fork so the community doesn't
> have to lose out.

All I want is to make sure you know your reasons and assumptions. The assumption there isn't a Phobos documentation with item-per-page was wrong. It seems to me referencing another module is trivial to fix as well.

Andrei

On 12/30/2015 08:32 PM, Adam D. Ruppe wrote:
> ~2010: I had just written this awesome dom.d library and wanted to
> document it and release it to the world. I write stuff like:
>
> /// Returns the text in the element. For example, innerText of
> <span>foo<span> is "foo" (without quotes)
> string innerText();
>
> And it came out mangled because of ddocs embedded HTML "feature" which
> doesn't encode the output. I proposed a fix, using ddoc's existing
> ESCAPES macro, and wrote the patch for dmd. Embedded html would be
> automatically encoded, but you can still define it in macro definitions,
> which can be inline with the comment, so it isn't hard to use when you
> want it.
>
> It was rejected. Walter didn't see what the problem was and I was told
> to just write $(LT)span$(GT)foo$(LT)/span$(GT). Seriously.

Could you please post a link to your proposed fix?

> The idea (and working program) was rejected because the team felt a
> post-processor was the wrong way to do it.

Do you have a link to that proposal and discussion?

> We got another website redesign in these years, and moved to dlang.org.
> The build process finally got documented after I and others complained
> for a while, but it was never actually cleaned up.

What steps do we need to take to clean the build process up?

> I just got sidetracked doing a quick paragraph fix for Andrei... and
> then wanted to see how far I could go. I got it working, despite the
> obstacles that ddoc's freeform macros bring, and surprise, it was
> rejected.

For reference: https://github.com/D-Programming-Language/dmd/pull/5319. I don't think it should have been accepted. It's not a good PR.

> Even the simpler patch that just collapses blank lines sits
> unanswered.

Sorry, the holiday period is not very productive what with the kids on vacation and whatnot. I'll pull your PR to my patch and will move forward with it. Thanks. Or did you work that into something standalone that I've missed?

> A tool to look for broken links sits unanswered. There's
> always a call to contributions, but when you do push through the painful
> process to get the code up (and the tester, the f$^%$^ing tester),
> nobody seems to care.

I agree and I'm sorry we're not moving faster with reviews, but really that's not ddo(c|x)'s fault.

> The combined experience over these last six years tells me that the
> website sucks and it keeps sucking because changing it sucks even more
> than using it. A new strategy was required.
>
> And here we are.

I hear your frustration, and for a good part I agree with it. Again: all I want is to make sure you have the right motivation and that you are fully informed of what's currently going on.


Andrei

On Thursday, 31 December 2015 at 02:40:17 UTC, Andrei Alexandrescu wrote:
> I agree and I'm sorry we're not moving faster with reviews, but really that's not ddo(c|x)'s fault.

Any chance you can respond to this question that I posted two days ago?

http://forum.dlang.org/post/pdjfvsmvdewwtjoicsam@forum.dlang.org

It doesn't require a review and only needs an answer of "yes" or "no". One of the nice things about Adam's site is that it has his email address on the front page.

On Thursday, 31 December 2015 at 01:32:56 UTC, Adam D. Ruppe wrote:

> The D leaders know how important examples are. We are often told adding more is low hanging fruit. I completely agree. But that's not ALL we need. He wants examples to get started, yes, but he also wants understanding to go beyond examples. That's where text helps. That's where understanding the function signatures help. Eventually, when people go to write their own libraries, they might want to do Phobos style genericity. You can't do that without understanding the function signatures... and those blobs of text are not understandable.

This is what hits me the most. Thats why we suggested "user contributed examples". PHP docs is the only place ive seen this. What is your stance on this and if you agree, how would you implement it? How would it work?

On 12/30/2015 10:07 PM, bachmeier wrote:
> On Thursday, 31 December 2015 at 02:40:17 UTC, Andrei Alexandrescu wrote:
>> I agree and I'm sorry we're not moving faster with reviews, but really
>> that's not ddo(c|x)'s fault.
>
> Any chance you can respond to this question that I posted two days ago?
>
> http://forum.dlang.org/post/pdjfvsmvdewwtjoicsam@forum.dlang.org
>
> It doesn't require a review and only needs an answer of "yes" or "no".
> One of the nice things about Adam's site is that it has his email
> address on the front page.

Yes, linking from the official site to the wiki is perfectly fine and recommended. Thanks! -- Andrei

On Thursday, 31 December 2015 at 06:39:27 UTC, Israel wrote:
> This is what hits me the most. Thats why we suggested "user contributed examples". PHP docs is the only place ive seen this. What is your stance on this and if you agree, how would you implement it? How would it work?

I do not agree with comments, but I do agree with making user submissions easy.

What I am envisioning right now is a kind of suggestion box thing on an individual page, where you can write up a bug report, an example, a comment, whatever, and have it emailed to me for further consideration. I'll do a quick fact-check editing then add it to the page.

Of course, you can also continue to submit pull requests through github (and I'd encourage this for bigger contributors), but just emailing your thoughts to me has a lower barrier of entry.

If we do a good job with the actual docs, you won't need comments. And better yet, you won't have to read through dozens of old ones to find the thing you need. On PHP, I find the comments list is too long and of too low of an average quality to be worth reading most the time.


The other important thing is being able to add new pages with ease. I just dreamed up this plan while in bed last night: a new page would be a D file with a huge leading comment and a bunch of documented unit tests. (Why a regular D file? Because then we can reference it by its name, it can be listed in the package listing automatically, and you can actually compile it to test the code without any extra extraction work!)

A comment at the top of "// just docs: Some Page Title" tells the generator to format it as a page rather than a normal module (this means the title in the comment will be used instead of the module name and it will have a table of contents of text sections instead of D classes, structs, and other members.)



I'll make a web service where you can upload your .d file and get an instant preview. This way, you can render it without needing to figure out how to install my doc generator and build the website on your own computer. Just submit the form.

Then, if you like it, you can either email the file to me and I'll take it from there, or submit a pull request to add your file.

The PR will only need to do:

git checkout -b my_new_file
git add my_new_file.d
git commit my_new_file.d -m 'add my new file!'
git push


No need to edit other macro files, build files, TOC files, whatever. My infrastructure takes care of that automatically. (via build **/*.d. It isn't rocket science and it baffles me that the official D system is so convoluted.)


What do the files look like? Well, I'm still working on nailing this down, but it is basically a hybrid of markdown and ddoc, but it does NOT support ddoc's custom macros. First off, I think they are a mess with everyone defining the same all over (look at Phobos, it gets confusing) and secondly it makes doing semantic analysis of the text harder since a ddoc macro is freeform. With unlimited macros, any $ or comma in the text might be significant syntax! That's a pain.

And it leads to bugs. Take a look at the References section here:

http://dlang.org/phobos/std_base64.html

It says "RFC 4648 - The Base16". The title is actually longer than that... it just had a comma in it and ddoc thought that meant end of argument.

There *are* ways to handle this in ddoc, of course, and they aren't even really hard. I deal with it every week in the TWID source. But it is just one little example of a bigger problem that limits how much automatic processing we can do. Not worth it.


Anyway, this is what contributed article file might look like. It continues to the end of this email:

// just docs: Introduction to my new doc system
/++

This is the intro line.

An intro paragraph. This is seen on the article listing too. Then leave two blank lines gap for the table of contents.


Now you start writing details. You can have headers, code examples, and so on like in any other doc.

$(H2 The headers look like ddoc macros)

They translate to the corresponding HTML and are parsed to form a table of contents which is automatically inserted in the blank area you left above.

$(H3 You can descend in with headers)

This section would be a sub-section of the H2 above.

$(H2 References and code)

You reference items with $(REF std.xml) and write code with `markdown style` or

---
ddoc style
---

More to come as it develops.

+/
module mydocs.intro; // you give it a D name to refer to in refs

/// you can also include documented unittests as full examples
/// that will be attached. I need to think of some way to
/// reference them in the text. If only D had named unittests!
unittest {
   // this code can actually be compiled with dmd -unittest
   // right out of the box to keep your examples working well.
}

// all done.

On Wednesday, 30 December 2015 at 04:03:42 UTC, Andrei Alexandrescu wrote:
> On 12/29/2015 09:09 PM, Adam D. Ruppe wrote:
>> Putting one item per page is far more important than I even realized
>> before getting into this.
>
> We already have that:
>
> https://dlang.org/library/std/array/join.html
>
> If I search for dlang array join that the third hit on google if I'm logged in, and the SECOND hit if I use an anonymous session that gives google no information about my searching habits. I hope you'll agree that renders the rest of your post moot, for which reason I afforded to snip it.

Which is a pity, because if you had read on you would have seen that Adam is aware of the existence of that separate page as he quoted the very link above in the part you snipped.

Frankly, the fact that the Google search taken as an example here returns two different pages on dlang.org, with no visible clues whether any of them is official or experimental, is very confusing to me. I had to search hard to figure out if and how they are linked together. Turns out the one with the separate page is experimental, and has been so for two years now, according to https://dlang.org/library/index.html#comment-1281452140.

> Adam, there's no nice way to put what follows. You can code a great deal, and I think the world of your engineering skills. But there is something to be said about a bias for action at the expense of strategy. I completely understand it's a lot more fun to start a project than to bring it to completion, but as they say in hardware, it's retired instructions that count. I wish you'd consider converting some of your myriad brilliant snippets into completed projects pushed into the standard distribution for prime time, and also (for this case) to consider strengthening the documentation tools we already have.

I greatly appreciate the energy both of you pour into D, but one accusing the other for not finishing projects is in this particular case a bit funny.

In my eyes there are three important aspects to quality documentation:
 1. Content
 2. Usability (legibility, X-ref, list of contents, indices, searching)
 3. Maintainability (ease to contribute, legibility in code, intuitive procedures)

The official documentation is sub-standard in 2 and 3, and Adam shows to be on a fast track to address these. It is a shame there is too much friction that this be done in the official documentation. But if it is too much for Adam, I am not encouraged to even try...

Bastiaan.