I wrote this manually as a mock of what I really want the docs to look like. I only marked up the top box and a wee bit of the bottom. The body text of the doc is written by us and is OK, but the function signature is a mess.

My mock:
http://arsdnet.net/mydoc.html

Verily, this is the status quo:
http://dlang.org/phobos/std_algorithm_searching.html#findSkip


First, you'll notice a big difference visually: the function signature is formatted in my example. The current thing is a mess of text.

I find one param per line to be much more readable, and this has precedent in professional documentation like Windows:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms632679%28v=vs.85%29.aspx


When you start adding more default argument stuff, the break up per line makes a huge, huge difference IMO.

Next, observe the highlighted words... and go ahead and try to click on them. Oh snap! They are links to the language features used. This is the web, let's link all over the place. I want to have relevant conceptual overviews available one click away on every function.

The "would you like to know more" bit isn't just a Starship Troopers film reference, but also a bit of important information (not actually written in the mock) about the idiom used there. I know D about as well as anyone, and I had to pause for a second to realize what that constraint *actually* meant. Let's just explain it in a link. Ideally, that'd be written by the function author in a section of the doc that can link in, but if nothing else, recognized patterns can link to generic idiom explanations.


Don't put that mouse down yet! Hover over the template args section. Behold the tooltip title. A little thing but maybe useful.

And then the coup de grace: mouse over "R1" or "haystack" in the definition... notice how it highlights the other occurrences of it?

Since the signature is incomplete without the template args, the runtime args, AND the constraint, I wanted some way to tie them all together visually.

They come in three parts of the signature and cannot be easily combined by space, but the mouse over highlighting can draw your eye's attention to them from out of the crowd of text.

Mouse over R1 and you instantly see it is the first param and referenced as a forward range in the constraint. Constraints are too loose to automate much more than this, but just seeing it there without having to swim through every word will help you tie it together in your squishy puny human brain.



There's a fair chunk of writing to actually realize the overviews, but the highlighting and formatting stuff can be done automatically. I've started work on a program, using Brian Schott's libdparse, to see what I can do to this end but I'm exceedingly busy with a million things.

And I am up two hours past bed time again, ttyl all.

On Friday, 25 December 2015 at 05:06:47 UTC, Adam D. Ruppe wrote:
> Next, observe the highlighted words... and go ahead and try to click on them. Oh snap! They are links to the language features used. This is the web, let's link all over the place. I want to have relevant conceptual overviews available one click away on every function.

Before I get to remarks about this demo, I'd like to connect the D doc situation with the current state of technical writing as I understand it(it's something I'm moving towards professionally). There's a set of trends I've identified in my survey of the landscape:

1. There's more code than ever and more of a need for code to be well-documented on a conceptual/hand-holding level, as well as the minimal "turn comments into docs" situation. "User manuals" are out - everyone wants easy UX - but "API docs" are in.
2. A shift away from proprietary tools like Word or Adobe Framemaker towards a heterogenous, open-format ecosystem - e.g. conversion tools like pandoc are important now.
3. "Topic-based authoring". This is mainly led by the DITA format, which imposes a particular *model* for how the documentation is laid out and how pieces of information may be related, vs. the more free-form linkage of a wiki or the linear text of traditional books. You typically see these topic-based systems appear in big enterprisey application software, but the general idea of fitting the text into a certain container is similar to the API doc tradition of following the class-and-method structure of the code.
4. A default approach that uses lighter markup like Markdown or wiki text in order to minimize the barriers to entry. "Heavy" formats like DocBook add friction to the prose-writing process, even though they're necessary once you pile on enough features. Authors may write in one and convert to the other.

The ideal documentation system for D addresses each of these - dedicated authors who are primarily adding prose text, a variety of output formats, models for defining code/data/textual relationships, and low friction inputs.

The mockup is good in that it cleans up the messy look and feel of the current presentation, and suggests the potential for interactivity. My main concern from a presentation standpoint is high latency, which in these interactive help systems typically comes in the form of slow page loads every time I try to click on something. My ideal system would stay one step ahead of me or allow me to stay on the same page as often as possible. Options for offline docs also usually fix the problem.

There is also a pretty good conceptual base in the existing ddoc system w/r to its goal of extending documentation comments. Its defaults are biased towards HTML presentation, which is OK, albeit not as trendy as Markdown.

What ddoc misses, though, is strong capability to invert the semantics towards prose-first - for example, if I wanted to write a very extensive tutorial that discusses a large codebase or how the codebase changes over time, then I'd write prose first, but reference an annotated version of the codebase so that I can call out specific parts of it. This is something that could be realized with a crude, hand-rolled system to extract the code and inline it, but it would most likely give up the semantic possibilities; I'd end up only having some syntax highlighting and not be able to let the reader click on the source and see the file in context, link function name callouts to their API documentation, etc.

One way to start seriously addressing all of these various issues is to export an annotated, pre-highlighted D source file with metadata, which could then be slurped up and processed again into an interactive API doc, a prose text, presentation slides, etc. This work can be seen as a two-parter: parsing the code and emitting useful, easily processed data - and then making default presentation of that data good, so that minimal effort is still rewarded. It's conceptually similar to the problem space of IDE integration, where different IDEs may want to query the source in different ways.

Semantic documentation source is also one of the rare situations where the XML ecosystem is at its best - XSLT basically exists to declaratively transform semantic markup into formatted presentation, as is done, for example, with DocBook. It wouldn't solve any *particular* problem to target XML, but it would expand people's ability to dream up and implement presentation-side solutions.

On Friday, 25 December 2015 at 05:06:47 UTC, Adam D. Ruppe wrote:
> I wrote this manually as a mock of what I really want the docs to look like. I only marked up the top box and a wee bit of the bottom. The body text of the doc is written by us and is OK, but the function signature is a mess.
>
> [...]

Why not take it to the next level while youre at it? Add user comments that can be rated by users and sorted by date.

On Friday, 25 December 2015 at 09:18:45 UTC, Israel wrote:
> Why not take it to the next level while youre at it? Add user comments that can be rated by users and sorted by date.

My dream does not include user comments. I don't think well-written documentation benefits from them, and adding such a thing would surely include some kind of speed hit on the page, harm downloadability, etc.; the page would then have a dynamically loaded section.

That isn't necessarily a dealbreaker, it just isn't a part of my dream.


On the other hand, I do dream of a suggestion box on the page that contacts the authors so they can incorporate it... which isn't totally different than comments.

So I'll put it down as a maybe.

On Friday, 25 December 2015 at 14:13:56 UTC, Adam D. Ruppe wrote:
> On Friday, 25 December 2015 at 09:18:45 UTC, Israel wrote:
>> Why not take it to the next level while youre at it? Add user comments that can be rated by users and sorted by date.
>
> My dream does not include user comments. I don't think well-written documentation benefits from them...

What happens when I see a DOC with comments is that sometimes the comments are more clear than the Doc itself, or there are tips or tricks that was not "well" documented.

Bubba.

On Friday, 25 December 2015 at 14:24:34 UTC, Bubbasaur wrote:
> On Friday, 25 December 2015 at 14:13:56 UTC, Adam D. Ruppe wrote:
>> On Friday, 25 December 2015 at 09:18:45 UTC, Israel wrote:
>>> Why not take it to the next level while youre at it? Add user comments that can be rated by users and sorted by date.
>>
>> My dream does not include user comments. I don't think well-written documentation benefits from them...
>
> What happens when I see a DOC with comments is that sometimes the comments are more clear than the Doc itself, or there are tips or tricks that was not "well" documented.
>
> Bubba.

Just one example that I said early, if you look here:

https://dlang.org/dmd-windows.html#switch-L (-Llinkerflag)

It's written:

"pass linkerflag to the linker link.exe , for example, -L/ma/li".


But at least on Windows, you need to put a space between -L and the PATH. Which It's weird, since with "-I" flag you don't need any space.

It took me 30 minutes until I find why my program wasn't compiling. Thanks to the Forum.

Bubba.

On Friday, 25 December 2015 at 07:41:11 UTC, James Hofmann wrote:
> 1. There's more code than ever and more of a need for code to be well-documented on a conceptual/hand-holding level

Absolutely.

> general idea of fitting the text into a certain container is similar to the API doc tradition of following the class-and-method structure of the code.

I recently went to add some explanation to std.base64 on how to get a string out of it since someone asked here. But I actually stopped on the official website just because I couldn't find a place to put it.

It didn't really fit in base64.d, the existing structure was all wrong and it was the wrong level. You can apply the concept using base64 but it isn't specific to it. It didn't fit on any of the existing pages. And adding a new page was just too painful for me. I ended up posting a couple quick sentences to the wiki - not even linked from anywhere - then running away. Everybody lost just because I felt blocked by the existing structure. (and I don't know wiki syntax, I felt like that was another blocker, and the dlang.org intralinking situation is terrible too)

This linked page is my dream for a narrow part of the docs: the function signature. The whitespace formatting can be worked into existing ddoc (I'm angry with existing ddoc and don't want to work with it for a while, but it isn't hugely difficult to do and I think will be a worthwhile incremental improvement), but my other vision is more revolutionary: docs that are written from a "how do I do this" viewpoint rather than a "how do I use this".


We still have the "how do I use this" sections, the existing pages basically fit that. But they feel written bottom-up. I want to do top down.

I've said before that I currently have a contractor working on some docs. My instructions for her were simply "write something cool in D and document it for future newbies". What she's written so far is already very different than what we have, though there's still a lot of work to do. (In particular, I'm kinda struggling with how much editing I want to do. The point is to document a newbie's journey as a guide to other newbies but I also see her making a few mistakes that I kinda want to correct... but she's learning quite a bit from those mistakes too and so could the readers. I hate tutorials that assume everything works. Users need to get used to seeing error messages and understanding both what they mean and why they are there.)

Once we get started, other things will follow. A friend of mine who is new to D just wrote something to save text to a google doc this week. The result is useful to him and his work and while it isn't super advanced D, the journey would be a good experience for others too. What libraries did he use? How did he overcome problems?



From these results-driven narratives, I can go back and start linking in more things - related narratives (generic D REST api tutorial, JSON use), language feature discussions (what was that nested function used in the callback? why use `is null` instead of `==`? what's `opIn`?) and idioms, concepts like string encoding and type system wrangling, and, of course, library docs for the used pieces.

Then link it back to the top level.



Of course, this stuff is a lot of work. I only have so much money I can spend on hiring people to work with this for me, and I have even less time to spend writing it myself... which is one reason why I'm now working on my new automation code.


> The mockup is good in that it cleans up the messy look and feel of the current presentation, and suggests the potential for interactivity. My main concern from a presentation standpoint is high latency, which in these interactive help systems typically comes in the form of slow page loads every time I try to click on something.

That's an issue of implementation. I'm literally in the same room as the server TWiD is on so I can't objectively judge its speed, but try clicking around the issues here:

http://arsdnet.net/this-week-in-d/dec-20.html

does it respond reasonably well? I made very little effort on optimization there... but also kept it simple and did all the processing ahead of time. The server hands out pre-built .html.gz files on demand, no dynamic parts.

My D doc search engine has dynamic parts: http://dpldocs.info/findSkip though I also wrote it with a bit of an eye toward efficiency because waiting for docs drives me nuts.

The static parts can all be downloaded in a bundle easily enough. Search could be a downloaded program too. Offline docs matter to me - I'm not always on the internet either!


> My ideal system would stay one step ahead of me or allow me to stay on the same page as often as possible.

Yes. I'm actually a wee bit split on the one page thing: I like having a long, single page with all the info. I just think there's a *lot* of advantages to it and actually consider this a strength of dlang.org (well, until std.algorithm got split up into separate modules. Now it is a pain, and I updated my dpldocs.info search engine in response)

Thought, one page per function is ok at times too.

> Semantic documentation source is also one of the rare situations where the XML ecosystem is at its best - XSLT basically exists to declaratively transform semantic markup into formatted presentation, as is done, for example, with DocBook. It wouldn't solve any *particular* problem to target XML, but it would expand people's ability to dream up and implement presentation-side solutions.

Yes, I really wish ddoc just output XML. I've tried to do it before using new macros (the default macros are unbelievably bad, there's actually decent semantics available through ddoc but they are all discarded in the macro phase) but it is a pain and still not as detailed as I like!

If you view source on my doc thing, you'll see the beginnings of semantic html. I want the identifiers to be called out! (There's data-ident="thing" in there now too.) I want the types to have metadata.

I'm tempted to make my thing spit out XML and then you'd do the rest with XSLT.... but I'm more familiar with html+css than I am with xslt so I'm just running with that.... yeah, personal limitation there. But semantic html isn't far off from xml anyway.

On Friday, 25 December 2015 at 14:24:34 UTC, Bubbasaur wrote:
>
> What happens when I see a DOC with comments is that sometimes the comments are more clear than the Doc itself, or there are tips or tricks that was not "well" documented.
>
> Bubba.

If the comments are more clear than the Doc, then they should _be_ the doc. Similarly, useful tips and tricks should probably be included as doc examples. In other words, I think we should encourage contribution to official documentation rather than dividing between official docs and user comments.

This does raise the bar to entry though, so maybe a suggestion box could be a nice middle ground.

On Friday, 25 December 2015 at 14:50:06 UTC, Adam D. Ruppe wrote:
> This linked page is my dream for a narrow part of the docs: the function signature. The whitespace formatting can be worked into existing ddoc (I'm angry with existing ddoc and don't want to work with it for a while, but it isn't hugely difficult to do and I think will be a worthwhile incremental improvement), but my other vision is more revolutionary: docs that are written from a "how do I do this" viewpoint rather than a "how do I use this".
>

Aren't these usually called tutorials? Or am I misunderstanding what you mean here?

>> My ideal system would stay one step ahead of me or allow me to stay on the same page as often as possible.
>
> Yes. I'm actually a wee bit split on the one page thing: I like having a long, single page with all the info. I just think there's a *lot* of advantages to it and actually consider this a strength of dlang.org (well, until std.algorithm got split up into separate modules. Now it is a pain, and I updated my dpldocs.info search engine in response)

This is literally one of the things that bothers me most about the documentation. Mostly when I try figure out how to do something I use google. Usually google picks out a couple of std module documentation pages for search queries I enter and then I have to Ctrl+F-guess my way through pages that make my scrollbar almost vanish or guess from function names (which are often somewhat abstract so it's hard to guess at what functions you should really check out). Then when you try to figure out if a function does what you want it to or not, often examples are missing and reasoning how to use it if it involves templates and template constraints isn't always straightforward if you aren't too familiar with the language and its idioms (ie: if you're new to it and trying to learn).

A single page really detailing what a function does and providing an example for how to use it is what I very much prefer.
I may also be a bit odd in this regard, since I use a pretty high zoom level in my browser, so scrolling down long pages (or navigations, such as the module list of std) is probably much more annoying to me than to everyone else (I often also want to hop through the documentation, like "oh okay I get how to use this function, but lets also check these two other things while Im here *scroll scroll scroll scroll*").
Plus I'm probably biased since I'm coming from C# and thus am heavily used to the MSDN docs.

> Thought, one page per function is ok at times too.

:-)

On Friday, 25 December 2015 at 15:51:02 UTC, rcorre wrote:
> On Friday, 25 December 2015 at 14:24:34 UTC, Bubbasaur wrote:
>>
>> What happens when I see a DOC with comments is that sometimes the comments are more clear than the Doc itself, or there are tips or tricks that was not "well" documented.
>>
>> Bubba.
>
> If the comments are more clear than the Doc, then they should _be_ the doc. Similarly, useful tips and tricks should probably be included as doc examples. In other words, I think we should encourage contribution to official documentation rather than dividing between official docs and user comments.
>
> This does raise the bar to entry though, so maybe a suggestion box could be a nice middle ground.

I agree, but you know in a perfect world nothing always work as we imagine, but anyway I think the suggestion box is a good idea.

Bubba.