Recently, I've been working on submitting a dub package of mine, sumtype, for inclusion in Phobos, D's standard library. Part of this process was converting sumtype's documentation comments from adrdox [1] syntax to ddoc [2] syntax.

However, when I ran ddoc on my updated module, I discovered something strange: some of the symbols that were supposed to show up in the documentation were missing. I hopped on over to issues.dlang.org to see if there were any relevant bug reports, and as I read through the search results [3], the awful truth began to dawn on me:

DDoc doesn't work.

I don't mean "its output is ugly," or "it lacks useful features," or "the macro syntax is awkward"--I mean that in a frankly staggering number of cases, DDoc flat-out fails to actually generate documentation for code with documentation comments. You know, the *one thing* it's supposed to do.

Here are some examples from bugzilla:

- DDoc ignores documentation comment that begins on the same line as the open curly brace
  https://issues.dlang.org/show_bug.cgi?id=5335

- Ddoc ignores methods in static else block
  https://issues.dlang.org/show_bug.cgi?id=11337

- Missing doc-comment for alias this, this(this)
  https://issues.dlang.org/show_bug.cgi?id=13069

- ddoc generator does not insert ddoc-ed unittest blocks for fromISOString, fromISOExtString, and fromSimpleString into the documentation
  https://issues.dlang.org/show_bug.cgi?id=16992

- Struct members missing from docs when the struct has a ditto ddoc comment
  https://issues.dlang.org/show_bug.cgi?id=18074

- Destructors and postblit constructors do not appear in DDoc output
  https://issues.dlang.org/show_bug.cgi?id=18860

- DDoc doesn't document symbols inside static foreach loops
  https://issues.dlang.org/show_bug.cgi?id=21399

- DDoc skips version else blocks inside templates
  https://issues.dlang.org/show_bug.cgi?id=21400

We ship this tool to every user of the D language. We advertise it as one of the quality-of-life features that helps D stand out from the crowd. And it doesn't work.

(And before you ask: no, ddox isn't any better.) [4]

Can someone explain to me why we haven't migrated everything to adrdox yet? I mean, say whatever you want about the look & feel, but at least it *actually generates documentation*.

[1] https://code.dlang.org/packages/adrdox
[2] https://dlang.org/spec/ddoc.html
[3] https://issues.dlang.org/buglist.cgi?keywords=ddoc&list_id=233822&resolution=---
[4] https://github.com/rejectedsoftware/ddox/issues?q=is%3Aissue+is%3Aopen+label%3A%22external+bug%22

On Mon, Nov 16, 2020 at 10:04:17PM +0000, Paul Backus via Digitalmars-d wrote: [...]
> DDoc doesn't work.
[...]
> - DDoc ignores documentation comment that begins on the same line as
> the open curly brace
>   https://issues.dlang.org/show_bug.cgi?id=5335

Wow, this one's been around for *that* long?!  Why isn't it fixed yet?


> - Ddoc ignores methods in static else block
>   https://issues.dlang.org/show_bug.cgi?id=11337

Yes, this is the problem with implementing the doc generator in the compiler: the static else is elided by the time the ddoc code looks at it, so it sees nothing and does nothing.

There is no easy fix, however. Potentially the static-if condition could be false for a good reason -- e.g., the code is obsolete and should not be documented. Or the static-if could be switching between two OSes / OS versions, in which case you want to document *both* branches (but how do you convey that in the generated docs?).


> - Missing doc-comment for alias this, this(this)
>   https://issues.dlang.org/show_bug.cgi?id=13069

Yeah, this one is a bona fide bug.


> - ddoc generator does not insert ddoc-ed unittest blocks for
> fromISOString, fromISOExtString, and fromSimpleString into the
> documentation
>   https://issues.dlang.org/show_bug.cgi?id=16992

This is another symptom of the way ddoc is implemented in the compiler: it expects all overloads to be declared consecutively. Break that assumption, and all sorts of weird things can happen.  It *should* be fixed, but might potentially require building ddoc.d from ground up. :-P (Not necessarily a bad thing, IMNSHO.)


> - Struct members missing from docs when the struct has a ditto ddoc comment
>   https://issues.dlang.org/show_bug.cgi?id=18074

The whole `/// ditto` thing is a hack, because there's an implicit assumption that you only apply it to symbols of like kind (like two functions, two structs, etc.).  When they're of two different kinds, weird things happen, like here.  It would be fun to find out what happens, for example, if you `/// ditto` together two completely disparate symbols, like a struct and a class, or an alias and a struct, etc..

What should the output be, for example, if the `/// ditto`'d symbols are both aggregates?  How should their members be formatted in the output? I don't think anybody even thought of this when ditto was implemented.


> - Destructors and postblit constructors do not appear in DDoc output
>   https://issues.dlang.org/show_bug.cgi?id=18860

Ouch.  Yeah, ddoc is a completely separate subsystem in the compiler, and I can see how when new features are added, the ddoc implications would be neglected.


> - DDoc doesn't document symbols inside static foreach loops
>   https://issues.dlang.org/show_bug.cgi?id=21399

ditto. ;-)


> - DDoc skips version else blocks inside templates
>   https://issues.dlang.org/show_bug.cgi?id=21400

I think it skips version else blocks in general. Probably the same bug as #11337.


> We ship this tool to every user of the D language. We advertise it as one of the quality-of-life features that helps D stand out from the crowd. And it doesn't work.
> 
> (And before you ask: no, ddox isn't any better.) [4]

Isn't ddox basically a better skin over ddoc? It follows the same parsing flawed assumptions that vanilla ddoc has.


> Can someone explain to me why we haven't migrated everything to adrdox yet?  I mean, say whatever you want about the look & feel, but at least it *actually generates documentation*.
[...]

I second this.  Adrdox fixes a lot of the flaws in the current ddoc implementation, and really should be the default rather than the current flawed version of ddoc.  (But I'm also anticipating an uphill fight to pry the ddoc macro system out of Walter's cold, dead, hands...)


T

-- 
He who does not appreciate the beauty of language is not worthy to bemoan its flaws.

On Monday, 16 November 2020 at 22:36:08 UTC, H. S. Teoh wrote:
> Yes, this is the problem with implementing the doc generator in the compiler
>
> There is no easy fix, however.

adrdox doesn't suffer from this.

Though it as its bugs in static if and version presentation, it *usually* manages to do the obvious thing, for example:

http://dpldocs.info/experimental-docs/arsd.simpledisplay.OpenGlShader.html

Just show the condition with the declaration (as you can see I actually turn `version(x) {} else {}` into just `version(!x)` since i think it is easier for human consumption here). The user can figure it out (hopefully).

> What should the output be, for example, if the `/// ditto`'d symbols are both aggregates?

http://dpldocs.info/experimental-docs/test2.foo.html

Not that hard, I didn't even consider this case specifically and it worked well enough!

What adrdox does is if it sees the ditto comment, it dittos the *comment*, not the output. So it acts like you copy/pasted the same comment text, but then reprocesses it for each decl separately.

Then, knowing it was ditto'd, it marks the two symbols as "related".

> Ouch.  Yeah, ddoc is a completely separate subsystem in the compiler, and I can see how when new features are added, the ddoc implications would be neglected.

Funny how adrdox manages to keep up with these things, but ddoc - despite (or perhaps BECAUSE of) being integrated in dmd doesn't.

On Tue, Nov 17, 2020 at 01:13:30AM +0000, Adam D. Ruppe via Digitalmars-d wrote:
> On Monday, 16 November 2020 at 22:36:08 UTC, H. S. Teoh wrote:
[...]
> > There is no easy fix, however.
[...]
> Just show the condition with the declaration (as you can see I
> actually turn `version(x) {} else {}` into just `version(!x)` since i
> think it is easier for human consumption here). The user can figure it
> out (hopefully).
[...]
> > What should the output be, for example, if the `/// ditto`'d symbols are both aggregates?
[...]
> What adrdox does is if it sees the ditto comment, it dittos the *comment*, not the output. So it acts like you copy/pasted the same comment text, but then reprocesses it for each decl separately.
[...]
> Funny how adrdox manages to keep up with these things, but ddoc - despite (or perhaps BECAUSE of) being integrated in dmd doesn't.

See, *this* is why we should replace dmd's built-in ddoc with adrdox. It's like superior in almost every way.  It's about time we swallowed our pride and admit that dmd's current ddoc implementation was a failure, and replace it with adrdox instead. It will do D so much good to fix this up.


T

-- 
I'm still trying to find a pun for "punishment"...

On Monday, 16 November 2020 at 22:04:17 UTC, Paul Backus wrote:
> Can someone explain to me why we haven't migrated everything to adrdox yet? I mean, say whatever you want about the look & feel, but at least it *actually generates documentation*.

I've seen adrdox also occasionally choke on certain kinds of input, and I feel we will see more cases of this if we were to begin using it more prominently. However:

The biggest real difference compared to DDoc/DDox is that adrdox is actively being maintained by a responsive and motivated developer. The upside is that Adam quickly fixes issues as they're reported (which is awesome). The downside is that, AFAIK, the project has a bus factor of one, which means that should Adam for whatever reason become unable to continue supporting it, we will go back to the exact same situation we are now with DDox (and perhaps possibly worse than DDoc, because in case of the latter, familiarity with the compiler translates somewhat to the ability to maintain DDoc).