More than two and half years ago, Sönke added ddox builds for the Phobos documentation. We all know that there are many reasons for ddox - being able to generate single pages for methods is just one, it also eliminates all the JavaScript hacks (e.g. the quickindex menu, anchors, ...) that we have added over time to deal with the shortcomings of ddoc.

This post originates from a recent discussion [2] that showed the higher ranking of the ddox pages in search engines because of those single pages, more static content and meta information.

To quote Adam [3]:

> ddox got a decent go up to here.
> But then we need to decide what's next - a clear goal, including a due date, gets us all aligned and removes a lot of the uncertainty on the author's side; it is some reassurance that they aren't wasting their time, and encourages outside teams to get onboard.

We got the MREF change into Phobos a month ago and Sönke has fixed the last blocking bug with ddox (broken source code links) a couple of days ago.

Imho it's quite impressive that he still pushes the project and as Adam
correctly said - we need to make a decision and have a clear deadline like 2.072 will be the last documentation build with ddoc, once it's released we will remove the ddoc Phobos build and make ddox (/library) the standard (with redirect, of course). This gives us also two to three months to test it properly again (it has been tested now for 2.5 years!!) and resolve issues if occurring.

Does this sound reasonable to everyone?

[1] https://github.com/dlang/dlang.org/pull/267
[2] http://forum.dlang.org/post/575079DE.2040508@erdani.org
[3] http://forum.dlang.org/post/zuzlvgqposlmtgdnxhmb@forum.dlang.org

On 06/04/2016 12:10 PM, Seb wrote:
> More than two and half years ago, Sönke added ddox builds for the Phobos
> documentation. We all know that there are many reasons for ddox - being
> able to generate single pages for methods is just one, it also
> eliminates all the JavaScript hacks (e.g. the quickindex menu, anchors,
> ...) that we have added over time to deal with the shortcomings of ddoc.
>
> This post originates from a recent discussion [2] that showed the higher
> ranking of the ddox pages in search engines because of those single
> pages, more static content and meta information.
>
> To quote Adam [3]:
>
>> ddox got a decent go up to here.
>> But then we need to decide what's next - a clear goal, including a due
>> date, gets us all aligned and removes a lot of the uncertainty on the
>> author's side; it is some reassurance that they aren't wasting their
>> time, and encourages outside teams to get onboard.
>
> We got the MREF change into Phobos a month ago and Sönke has fixed the
> last blocking bug with ddox (broken source code links) a couple of days
> ago.
>
> Imho it's quite impressive that he still pushes the project and as Adam
> correctly said - we need to make a decision and have a clear deadline
> like 2.072 will be the last documentation build with ddoc, once it's
> released we will remove the ddoc Phobos build and make ddox (/library)
> the standard (with redirect, of course). This gives us also two to three
> months to test it properly again (it has been tested now for 2.5
> years!!) and resolve issues if occurring.
>
> Does this sound reasonable to everyone?
>
> [1] https://github.com/dlang/dlang.org/pull/267
> [2] http://forum.dlang.org/post/575079DE.2040508@erdani.org
> [3] http://forum.dlang.org/post/zuzlvgqposlmtgdnxhmb@forum.dlang.org

I recall there were a few issues with ddox rendering up until relatively recently, have all been fixed?

I don't see these options mutually exclusive, though I do want to make ddox the default/preferred rendering if ready. Consider e.g. https://www.gnu.org/software/make/manual/ which offers the documentation in various formats, including fewer/multiple pages.

Now, I gave myself a few minutes to take a look at ddox:

* http://dlang.org/phobos/ looks nicer than http://dlang.org/library/. The intro text is not very important, but the grouping is nice. The table vertical split ratio is also nicer.

* I assume the dead links matter will be solved.

* The cheat sheet made sense for the single-page docs but not for the new ones. Consider e.g. http://dlang.org/library/std/algorithm/comparison.html - it's two tables with the same row headings one after another. The information should be consolidated in one table (I'm speaking from the user's perspective; I know it's hard).

* There are a few formatting issues that I like less in the new docs. These are of course fixable with relative ease. Looking at e.g. http://dlang.org/library/std/algorithm/comparison/among.html:

** the vertical spacing is generally too fluffy (do we really need 25% of the page occupied by Authors and License for _every_ function?)

** Do we need the actual "Prototypes" heading or is it implicit?

** The booktable style is nice for tables that span the entire page width; for parameters the boxes at http://dlang.org/phobos/std_algorithm_comparison.html#among seem nicer to me.

** Do we need the actual heading "Parameters", or it suffices to put "Parameter" instead of "Name" in the parameter box heading?

** Examples are nice but again vertical spacing overall and between consecutive examples is too large.

** Generally the sentiment of the formatting is "too much foreplay". Get to the point already, don't have people scroll through fluff. Relegate Authors and License to a small footnote below everything - even after disqus.

* The hotlinks to parameters (e.g. "value", "values") seem unnecessary unless the synopsis is very long which shouldn't be the case. It's also inconsistent - pred is not hotlinked.

* How are we going to deal with links (possibly with hashes) pointing to the existing artifacts? Again the simplest way is to keep the old docs but just deemphasize them on dlang.org.

* I'd like a "Show entire module documentation as single page" link for e.g. http://dlang.org/library/std/algorithm/comparison.html.

* How do we address the ebook/pdf production? Do we continue using ddoc or look into a ddox approach?

* Any other extant issues with /library/ (I recall there were a few cases in which not all text was rendered)?


Andrei

On 06/04/2016 07:01 PM, Andrei Alexandrescu wrote:
> * The cheat sheet made sense for the single-page docs but not for the
> new ones. Consider e.g.
> http://dlang.org/library/std/algorithm/comparison.html - it's two tables
> with the same row headings one after another. The information should be
> consolidated in one table (I'm speaking from the user's perspective; I
> know it's hard).

I have a pull request open that addresses the issue:

https://github.com/dlang/phobos/pull/4397

It hides the handcrafted cheat sheets. It's not been pulled yet, because the handcrafted cheat sheets may be better than the generated ones.

I haven't been able to figure out how to hide the generated ones. Also, the handcrafted ones tend to get outdated which is especially bad with DDOX, because you can't just scroll down to it.

Am 04.06.2016 um 19:18 schrieb ag0aep6g:
> On 06/04/2016 07:01 PM, Andrei Alexandrescu wrote:
>> * The cheat sheet made sense for the single-page docs but not for the
>> new ones. Consider e.g.
>> http://dlang.org/library/std/algorithm/comparison.html - it's two tables
>> with the same row headings one after another. The information should be
>> consolidated in one table (I'm speaking from the user's perspective; I
>> know it's hard).
>
> I have a pull request open that addresses the issue:
>
> https://github.com/dlang/phobos/pull/4397
>
> It hides the handcrafted cheat sheets. It's not been pulled yet, because
> the handcrafted cheat sheets may be better than the generated ones.
>
> I haven't been able to figure out how to hide the generated ones. Also,
> the handcrafted ones tend to get outdated which is especially bad with
> DDOX, because you can't just scroll down to it.

I think ideally we could do something like introducing a special "Cheatsheet" section or macro for each function that DDOX could aggregate or use instead of the normal short description. Also interesting (although less so since we started using sub modules) would be to do the same for a group/category that would be used to group symbols by other means that simply their type category.

The downside is that the normal Ddoc generated docs couldn't use that information to display the overview tables like it does now, so it would make most sense to do this after making the switch.

On 06/04/2016 07:25 PM, Sönke Ludwig wrote:
> I think ideally we could do something like introducing a special
> "Cheatsheet" section or macro for each function that DDOX could
> aggregate or use instead of the normal short description. Also
> interesting (although less so since we started using sub modules) would
> be to do the same for a group/category that would be used to group
> symbols by other means that simply their type category.
>
> The downside is that the normal Ddoc generated docs couldn't use that
> information to display the overview tables like it does now, so it would
> make most sense to do this after making the switch.

Thinking about this, if the information in the cheat sheet is the most helpful, maybe we should be putting exactly that in the summaries of things. Meaning, maybe we should change summaries to be one short sentence and/or one short example.

Am 04.06.2016 um 19:01 schrieb Andrei Alexandrescu:
>
> I recall there were a few issues with ddox rendering up until relatively
> recently, have all been fixed?

I think they have. Vladimir has reported a bunch of them over time and all of those have been fixed.

> I don't see these options mutually exclusive, though I do want to make
> ddox the default/preferred rendering if ready. Consider e.g.
> https://www.gnu.org/software/make/manual/ which offers the documentation
> in various formats, including fewer/multiple pages.
>
> Now, I gave myself a few minutes to take a look at ddox:
>
> * http://dlang.org/phobos/ looks nicer than http://dlang.org/library/.
> The intro text is not very important, but the grouping is nice. The
> table vertical split ratio is also nicer.

This is currently just the generic module overview. We could simply replace it by a handcrafted version. Shall I copy the one from /phobos/? ...or I could also just read the corresponding .ddoc file and render that.

> * I assume the dead links matter will be solved.
>
> * The cheat sheet made sense for the single-page docs but not for the
> new ones. Consider e.g.
> http://dlang.org/library/std/algorithm/comparison.html - it's two tables
> with the same row headings one after another. The information should be
> consolidated in one table (I'm speaking from the user's perspective; I
> know it's hard).

Leaving that to the other sub thread.

>
> * There are a few formatting issues that I like less in the new docs.
> These are of course fixable with relative ease. Looking at e.g.
> http://dlang.org/library/std/algorithm/comparison/among.html:
>
> ** the vertical spacing is generally too fluffy (do we really need 25%
> of the page occupied by Authors and License for _every_ function?)

I intended to tackle that multiple times. Definitely agreed that it should be much smaller. On the plus side, it's placed after all of the real meat of the page (except for the comments), so it doesn't really do any harm.

>
> ** Do we need the actual "Prototypes" heading or is it implicit?

That's definitely debatable. The first special purpose sections could be layed out in a few different ways:

	Primary heading

	brief description

	[prototypes]

	detailed description

	Parameters and other sections

or maybe

	Primary heading

	brief description
	detailed description

	[prototypes]
	[parameters]

or maybe

	Primary heading

	brief description

	[prototypes]
	[parameters]

	detailed description

They all have their pros and cons, and those also depend on what is displayed on the parent page. If, for example, the parent page already displays the prototypes (which I intended to add in some form), moving them further down below the page can make sense. Otherwise it can be good to have them near the top, so that the reader gets an early impression about the function, before reading through the detailed description.

> ** The booktable style is nice for tables that span the entire page
> width; for parameters the boxes at
> http://dlang.org/phobos/std_algorithm_comparison.html#among seem nicer
> to me.

While I don't particularly like those either (but I'll leave the design discussion to other people) - I can of course add that right away. I just used the default table style there and simply didn't notice the style change in /phobos/.

> ** Do we need the actual heading "Parameters", or it suffices to put
> "Parameter" instead of "Name" in the parameter box heading?

Depending on the general page layout we should definitely be able to drop the heading.

> ** Examples are nice but again vertical spacing overall and between
> consecutive examples is too large.

Partially agreed, the spacing could indeed be reduced a bit, however, one of the big advantages of the single-page layout is that we can use more vertical space to give the page more visual structure and thus make it easier to scan for a particular section.

> ** Generally the sentiment of the formatting is "too much foreplay". Get
> to the point already, don't have people scroll through fluff. Relegate
> Authors and License to a small footnote below everything - even after
> disqus.

Authors+License sure, but the rest is content, right? Or what else would you consider foreplay?

> * The hotlinks to parameters (e.g. "value", "values") seem unnecessary
> unless the synopsis is very long which shouldn't be the case. It's also
> inconsistent - pred is not hotlinked.

You are probably right. Linking to template arguments is currently not supported, so dropping parameter linking in general is definitely the more comfortable change ;-)

> * How are we going to deal with links (possibly with hashes) pointing to
> the existing artifacts? Again the simplest way is to keep the old docs
> but just deemphasize them on dlang.org.

Ideally we'd set up redirection rules to forward to the new pages, but page anchors will be problematic. One idea to solve that would be, instead of defining server-side redirection rules, to put some JavaScript code inside of the original /phobos/ docs that detects the active page anchor and redirects to the proper /library/ sub page.

> * I'd like a "Show entire module documentation as single page" link for
> e.g. http://dlang.org/library/std/algorithm/comparison.html.

So, conceptually a link to /phobos/? What's the motivation for that? Can we solve that by adding more information to the module page, possibly with a button to show that on-demand?

> * How do we address the ebook/pdf production? Do we continue using ddoc
> or look into a ddox approach?

I'd say we stay with pure Ddoc as long as that is sufficient. It may get interesting depending on which way we go with the cheatsheets/overview tables.

> * Any other extant issues with /library/ (I recall there were a few
> cases in which not all text was rendered)?

As stated, I think all of the known DDOX/dpl-docs related issues have been solved. However, there are still some issues with the JSON output of DMD (alias names are not preserved, no documentation inside of `static if` and some others). DDOX also has a doc parser based on libdparse, which could be a good alternative to work around that. I'll have a look at that.

BTW, thanks for taking the time to write this down (again)! I think this has also been one of the main issues in the process that there was generally very few feedback, I guess mostly because /library/ was quite hidden in the "resources" section.

What I think we should do with these issues is to put them into two categories: Needs to be solved _before_ or can be solved _after_ the switch. Especially those that are present in /phobos/ now are candidates for the latter category.

On Saturday, 4 June 2016 at 16:10:14 UTC, Seb wrote:
> Imho it's quite impressive that he still pushes the project and as Adam
> correctly said - we need to make a decision and have a clear deadline like 2.072 will be the last documentation build with ddoc, once it's released we will remove the ddoc Phobos build and make ddox (/library) the standard (with redirect, of course). This gives us also two to three months to test it properly again (it has been tested now for 2.5 years!!) and resolve issues if occurring.

One wholesome switch is probably too harsh. We might discover new issues only after the switch. Instead it could be done gradually, in this order:

1. /library/ is promoted as the primary Phobos documentation in the site navigation.
2. /phobos/ is removed from search indexing.
3. /phobos/ is removed from site navigation.
4. /phobos/ is removed.

We could start with step 1 upon 2.072's release.

On 6/4/16 2:23 PM, Sönke Ludwig wrote:
> Am 04.06.2016 um 19:01 schrieb Andrei Alexandrescu:
>>
>> I recall there were a few issues with ddox rendering up until relatively
>> recently, have all been fixed?
>
> I think they have. Vladimir has reported a bunch of them over time and
> all of those have been fixed.

Great, thanks.

>> * http://dlang.org/phobos/ looks nicer than http://dlang.org/library/.
>> The intro text is not very important, but the grouping is nice. The
>> table vertical split ratio is also nicer.
>
> This is currently just the generic module overview. We could simply
> replace it by a handcrafted version. Shall I copy the one from /phobos/?
> ...or I could also just read the corresponding .ddoc file and render that.

Any way you prefer. I'd be fine with copying the manually-crafted table from /phobos/, and eliminate or modify the short text preceding it.

>> ** the vertical spacing is generally too fluffy (do we really need 25%
>> of the page occupied by Authors and License for _every_ function?)
>
> I intended to tackle that multiple times. Definitely agreed that it
> should be much smaller. On the plus side, it's placed after all of the
> real meat of the page (except for the comments), so it doesn't really do
> any harm.

Cool, thanks. (I think the comments will play a nice role in the future.)

>> ** Do we need the actual "Prototypes" heading or is it implicit?
>
> That's definitely debatable. The first special purpose sections could be
> layed out in a few different ways:

Oh, I meant the actual word. "Prototypes". It takes a lot of vertical space and doesn't add information, because obviously what follows are the prototypes.

>> ** The booktable style is nice for tables that span the entire page
>> width; for parameters the boxes at
>> http://dlang.org/phobos/std_algorithm_comparison.html#among seem nicer
>> to me.
>
> While I don't particularly like those either (but I'll leave the design
> discussion to other people) - I can of course add that right away. I
> just used the default table style there and simply didn't notice the
> style change in /phobos/.

Makes sense. Any reasonable design should work fine there.

>> ** Do we need the actual heading "Parameters", or it suffices to put
>> "Parameter" instead of "Name" in the parameter box heading?
>
> Depending on the general page layout we should definitely be able to
> drop the heading.

Awesome.

>> ** Examples are nice but again vertical spacing overall and between
>> consecutive examples is too large.
>
> Partially agreed, the spacing could indeed be reduced a bit, however,
> one of the big advantages of the single-page layout is that we can use
> more vertical space to give the page more visual structure and thus make
> it easier to scan for a particular section.

Reasonable.

>> ** Generally the sentiment of the formatting is "too much foreplay". Get
>> to the point already, don't have people scroll through fluff. Relegate
>> Authors and License to a small footnote below everything - even after
>> disqus.
>
> Authors+License sure, but the rest is content, right? Or what else would
> you consider foreplay?

I'm thinking just rendering the information in an obvious and immediate manner that doesn't require scrolling or eyes to jump across the page. Consider e.g. https://doc.rust-lang.org/std/boxed/trait.FnBox.html. You open the page and boom, you know it's a trait, you see the signature (without the need to read "Hey hey hey...! here comes the signature!" etc), you see a small yet conspicuous colored box with important information about instability, you see the synopsys, and the example is right after. No need to scroll! It's all clean, immediate, and matter-of-fact, no pomp and circumstance. Imagine that rendered with all components delineated by own headings "<br><br><br>Warning:<br><br>Unstable (fnbox #28796): Newly introduced<br><br><br>" etc.

>> * The hotlinks to parameters (e.g. "value", "values") seem unnecessary
>> unless the synopsis is very long which shouldn't be the case. It's also
>> inconsistent - pred is not hotlinked.
>
> You are probably right. Linking to template arguments is currently not
> supported, so dropping parameter linking in general is definitely the
> more comfortable change ;-)

Noice.

>> * How are we going to deal with links (possibly with hashes) pointing to
>> the existing artifacts? Again the simplest way is to keep the old docs
>> but just deemphasize them on dlang.org.
>
> Ideally we'd set up redirection rules to forward to the new pages, but
> page anchors will be problematic. One idea to solve that would be,
> instead of defining server-side redirection rules, to put some
> JavaScript code inside of the original /phobos/ docs that detects the
> active page anchor and redirects to the proper /library/ sub page.

Seems we can easily defer that if we continue to produce both forms.

>> * I'd like a "Show entire module documentation as single page" link for
>> e.g. http://dlang.org/library/std/algorithm/comparison.html.
>
> So, conceptually a link to /phobos/? What's the motivation for that? Can
> we solve that by adding more information to the module page, possibly
> with a button to show that on-demand?

I want to collect statistics on how many people continue to use the one-page style. If too few, we can remove that later.

This is no competition and no zero sum game: we have two formats and we can leverage that. I and many others prefer page-per-artifact, so /library/ is awesome. However there might be a bunch of folks out there comfy with the page-per-module, and why hurt them if we needn't.

>> * How do we address the ebook/pdf production? Do we continue using ddoc
>> or look into a ddox approach?
>
> I'd say we stay with pure Ddoc as long as that is sufficient. It may get
> interesting depending on which way we go with the cheatsheets/overview
> tables.

OK.

>> * Any other extant issues with /library/ (I recall there were a few
>> cases in which not all text was rendered)?
>
> As stated, I think all of the known DDOX/dpl-docs related issues have
> been solved. However, there are still some issues with the JSON output
> of DMD (alias names are not preserved, no documentation inside of
> `static if` and some others). DDOX also has a doc parser based on
> libdparse, which could be a good alternative to work around that. I'll
> have a look at that.

Thanks.

> BTW, thanks for taking the time to write this down (again)! I think this
> has also been one of the main issues in the process that there was
> generally very few feedback, I guess mostly because /library/ was quite
> hidden in the "resources" section.
>
> What I think we should do with these issues is to put them into two
> categories: Needs to be solved _before_ or can be solved _after_ the
> switch. Especially those that are present in /phobos/ now are candidates
> for the latter category.

Sounds good. I have an honest answer and a manager answer. The honest answer is that only the broken links need to be fixed before. The manager answer is that everything should be fixed before, because then the release creates a forcing function. Otherwise, it could take months before any is looked at.


Andrei

On 6/4/16 2:33 PM, Vladimir Panteleev wrote:
> On Saturday, 4 June 2016 at 16:10:14 UTC, Seb wrote:
>> Imho it's quite impressive that he still pushes the project and as Adam
>> correctly said - we need to make a decision and have a clear deadline
>> like 2.072 will be the last documentation build with ddoc, once it's
>> released we will remove the ddoc Phobos build and make ddox (/library)
>> the standard (with redirect, of course). This gives us also two to
>> three months to test it properly again (it has been tested now for 2.5
>> years!!) and resolve issues if occurring.
>
> One wholesome switch is probably too harsh. We might discover new issues
> only after the switch. Instead it could be done gradually, in this order:
>
> 1. /library/ is promoted as the primary Phobos documentation in the site
> navigation.
> 2. /phobos/ is removed from search indexing.
> 3. /phobos/ is removed from site navigation.
> 4. /phobos/ is removed.
>
> We could start with step 1 upon 2.072's release.

Sounds good to me, thanks. Delegation/lieutenantship/empowering for the win. I think we should also secure Martin's buy-in to make sure. -- Andrei

On 04/06/16 18:10, Seb wrote:

> We got the MREF change into Phobos a month ago and Sönke has fixed the
> last blocking bug with ddox (broken source code links) a couple of days
> ago.

I found a minor issue recently. If there's more than one symbol in the same module with the same name but with different casing, all these symbols are shown on the same "single symbol page". Not sure if that's solvable due to some operating systems not using case sensitive file systems.

-- 
/Jacob Carlborg