On 12/30/2013 1:31 PM, Jacob Carlborg wrote:
> On 2013-12-30 21:58, Walter Bright wrote:
>> I'll have to disagree on that. I just finished doing it with
>> std.datetime, and it didn't take more than a few minutes with global
>> query-search-replace.
>
> How did you limit that search to only the documentation?

The "query" part, meaning it highlighted each one and asked for yes/no.

http://www.gnu.org/software/emacs/manual/html_node/emacs/Query-Replace.html

On 12/30/2013 1:25 PM, Sönke Ludwig wrote:
> Identifiers in documentation comments that are function parameters or
> are names that are in scope at the associated declaration are emphasized
> in the output.
>
> So the same problem is already reality - you already have to go through
> the documentation to see if there are any mis-highlighted words.

1. having a bad idea is not justification for doing it again :-)

2. the requirement that the name be "in scope" does a pretty good job of reducing the problem. But cross references, are by definition CROSS references, i.e. they cross scopes. The whole point of them is they cross scopes, which means this becomes a much bigger issue.

Am 30.12.2013 23:13, schrieb Walter Bright:
> On 12/30/2013 1:25 PM, Sönke Ludwig wrote:
>> Identifiers in documentation comments that are function parameters or are names that are in scope at the associated declaration are emphasized in the output.
>>
>> So the same problem is already reality - you already have to go through the documentation to see if there are any mis-highlighted words.
> 
> 1. having a bad idea is not justification for doing it again :-)

Of course not, but my point was that the mistake has already been made, so we may as well make the best out of it... not really doing it again in that sense.

But if the decision is that in fact this was a mistake, it should get removed and replaced by a proper solution sooner rather than later (it's not that bad to lose some highlighting and Phobos uses $(D ...) anyway, so just removing it wouldn't hurt much). And in that regard I'd like to point to "D's goals for embedded documentation" <http://dlang.org/ddoc.htm>. I very much agree with them, but the current reality looks like the opposite - lots of macro-pseudo-HTML markup, very difficult to read unprocessed in many places. So something like using "#" or "()" to mark symbols + automatic cross referencing for marked symbols would be a *really* nice to have.

> 
> 2. the requirement that the name be "in scope" does a pretty good job of reducing the problem. But cross references, are by definition CROSS references, i.e. they cross scopes. The whole point of them is they cross scopes, which means this becomes a much bigger issue.
> 

This is how I see it: single identifiers are only searched in the current scope and in parent scopes of the same module. Cross references to foreign scopes need to use qualified names (either ClassName.member or pack.mod.Symbol). This avoids any inter-module issues and also a lot of intra-module issues. Symbols in code blocks on the other hand can be cross referenced completely automatically according to the usual D lookup rules (including imports).

On 12/30/2013 3:13 PM, Sönke Ludwig wrote:
> Am 30.12.2013 23:13, schrieb Walter Bright:
>> On 12/30/2013 1:25 PM, Sönke Ludwig wrote:
>>> Identifiers in documentation comments that are function parameters or
>>> are names that are in scope at the associated declaration are emphasized
>>> in the output.
>>>
>>> So the same problem is already reality - you already have to go through
>>> the documentation to see if there are any mis-highlighted words.
>>
>> 1. having a bad idea is not justification for doing it again :-)
>
> Of course not, but my point was that the mistake has already been made,
> so we may as well make the best out of it... not really doing it again
> in that sense.
>
> But if the decision is that in fact this was a mistake, it should get
> removed and replaced by a proper solution sooner rather than later (it's
> not that bad to lose some highlighting and Phobos uses $(D ...) anyway,
> so just removing it wouldn't hurt much).

Ddoc is very heavily used. I don't believe we can make such silent, breaking changes.


> And in that regard I'd like to
> point to "D's goals for embedded documentation"
> <http://dlang.org/ddoc.htm>. I very much agree with them, but the
> current reality looks like the opposite - lots of macro-pseudo-HTML
> markup, very difficult to read unprocessed in many places. So something
> like using "#" or "()" to mark symbols + automatic cross referencing for
> marked symbols would be a *really* nice to have.

I don't see any particular advantage to using (name) rather than $(REF name), and some serious disadvantages with false positives like func(param). Oops! A markup language should really strive to minimize special syntax. (I'm often tripped up by false positives in the wacky markup languages in Skype, Github, Reddit, etc., each of which feels compelled to reinvent markup.)


> > 2. the requirement that the name be "in scope" does a pretty good job of
>> reducing the problem. But cross references, are by definition CROSS
>> references, i.e. they cross scopes. The whole point of them is they
>> cross scopes, which means this becomes a much bigger issue.
>>
>
> This is how I see it: single identifiers are only searched in the
> current scope and in parent scopes of the same module. Cross references
> to foreign scopes need to use qualified names (either ClassName.member
> or pack.mod.Symbol). This avoids any inter-module issues and also a lot
> of intra-module issues. Symbols in code blocks on the other hand can be
> cross referenced completely automatically according to the usual D
> lookup rules (including imports).

My bias is towards being extremely conservative in adding new features to Ddoc, instead we should strive to get the max out of the existing system, and even rely on external post processors.

Am 31.12.2013 00:37, schrieb Walter Bright:
> On 12/30/2013 3:13 PM, Sönke Ludwig wrote:
>> Am 30.12.2013 23:13, schrieb Walter Bright:
>>> On 12/30/2013 1:25 PM, Sönke Ludwig wrote:
>>>> Identifiers in documentation comments that are function parameters or
>>>> are names that are in scope at the associated declaration are
>>>> emphasized
>>>> in the output.
>>>>
>>>> So the same problem is already reality - you already have to go through the documentation to see if there are any mis-highlighted words.
>>>
>>> 1. having a bad idea is not justification for doing it again :-)
>>
>> Of course not, but my point was that the mistake has already been made, so we may as well make the best out of it... not really doing it again in that sense.
>>
>> But if the decision is that in fact this was a mistake, it should get removed and replaced by a proper solution sooner rather than later (it's not that bad to lose some highlighting and Phobos uses $(D ...) anyway, so just removing it wouldn't hurt much).
> 
> Ddoc is very heavily used. I don't believe we can make such silent, breaking changes.
> 

It's just word highlighting! Breaking changes in the language are one thing, but this kind of fix is a whole different category. Keeping it will only make the problem worse over time. If there is a reasonable deprecation path, that would be better, of course, but the current situation is the worst of all possibilities.

> 
>> And in that regard I'd like to
>> point to "D's goals for embedded documentation"
>> <http://dlang.org/ddoc.htm>. I very much agree with them, but the
>> current reality looks like the opposite - lots of macro-pseudo-HTML
>> markup, very difficult to read unprocessed in many places. So something
>> like using "#" or "()" to mark symbols + automatic cross referencing for
>> marked symbols would be a *really* nice to have.
> 
> I don't see any particular advantage to using (name) rather than $(REF
> name), and some serious disadvantages with false positives like
> func(param). Oops! A markup language should really strive to minimize
> special syntax. (I'm often tripped up by false positives in the wacky
> markup languages in Skype, Github, Reddit, etc., each of which feels
> compelled to reinvent markup.)

It's not "(name)", but "name()" or "name(param)", which is recognized as
a function call. "#name" also forces recognition as an identifier, as
well as "::name" in C++'s case.

I agree that going overboard with special syntax is a bad idea, but symbols in a code documentation language is such a fundamental thing that it should surely be among the first things, if not *the* first thing, that deserves a concise syntax.

Also, the official first four goals of DDOC are:

> 1. It looks good as embedded documentation, not just after it is
>    extracted and processed.
> 2. It's easy and natural to write, i.e. minimal reliance on <tags> and
>    other clumsy forms one would never see in a finished document.
> 3. It does not repeat information that the compiler already knows from
>    parsing the code.
> 4. It doesn't rely on embedded HTML, as such will impede extraction
>    and formatting for other purposes.

Using $(XREF), $(TABLE $(TR $(TD ...))), $(LINK a b) $(D ...) and so on
is nothing but markup. If we go that route, we could as well remove all
syntax, except for the macros:

$(SUMMARY ...)
$(SECTION See_Also
   ...
)
$(PARAMS
  $(PARAM x ...)
  $(PARAM y ...)
)
$(SECTION Example
  $(CODEBLOCK
    ...
  )
)

But you added the syntax for a reason back then (readability ...and writability). And the same goes for the auto-detection of identifiers. I think the only problem is that the default would have been better off the other way around.

On 12/30/13 1:00 PM, Walter Bright wrote:
> On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
>> Oh the default is to link and then disable manually? That's still
>> manual, just
>> the default is different :o). I can work with that.
>
> As I mentioned elsewhere, I dislike the requirement to use the inverse
> feature. Consider maintenance - I add a symbol, and then have to grep
> the rest of the module for any inadvertent collisions? Features should
> be positive ones, not negative.

Your arguments are convincing.

Andrei

Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:
> On 12/30/13 1:00 PM, Walter Bright wrote:
>> On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
>>> Oh the default is to link and then disable manually? That's still
>>> manual, just
>>> the default is different :o). I can work with that.
>>
>> As I mentioned elsewhere, I dislike the requirement to use the inverse feature. Consider maintenance - I add a symbol, and then have to grep the rest of the module for any inadvertent collisions? Features should be positive ones, not negative.
> 
> Your arguments are convincing.
> 
> Andrei
> 

The argument in isolation is sound, no doubt. But alone it doesn't lead to a solution*. There are a lot of arguments regarding the current situation and if the decision is not to use the existing _negative rule, then all of those should be carefully taken into account.

 - Visual noise and effort when using $(XREF) style macros
 - $(XREF) isn't sufficient! A multitude of macros is actually needed
 - There is no standard set of macros, it's project specific
 -> The whole documentation needs to be revised when moving code between
    projects!
 - Macros need to be carefully crafted so that they don't encode
   knowledge about the final documentation/file structure
 - Macros alone can't cope with nested directory structures
 - The cognitive load for multiple such macros would very likely cause
   them to go unused in many cases

My favorite solution would be to introduce a simple #identifier.chain or @identifier.chain syntax to let the compiler or documentation generator insert the proper links or macro calls. Using a sepecial $(REF identifier.chain) pseudo macro, like Jacob suggested, would work as well, but I can't see how the additional syntax/visual/typing overhead is justified for such a common feature - at least when trying to find an optimal solution, compared to the current macro based system it would of course still be a big improvement.

But still, considering that there is precedence for the _negative approach (i.e. the problems of it are already there), I find using that acceptable, too, even if the concept is not as pretty/clean.


* *Some* solution is needed. The different structure of the DDOX documentation for example makes the $(XREF) macros worthless, so they are currently completely disabled and automatic link generation is used instead.

Am Tue, 31 Dec 2013 00:13:41 +0100
schrieb Sönke Ludwig <sludwig@outerproduct.org>:

> [...] I very much agree with them, but the
> current reality looks like the opposite - lots of macro-pseudo-HTML
> markup, very difficult to read unprocessed in many places. So something
> like using "#" or "()" to mark symbols + automatic cross referencing for
> marked symbols would be a *really* nice to have.

I am VERY much in favor of a more MarkDown style syntax and # for identifier references! It is also very verbose to create bullet-point lists and you always have to double-check you don't forget to replace all ) with $(RPAREN). If you misspell anything it is lost in the output.

-- 
Marco

On Tuesday, 31 December 2013 at 09:29:20 UTC, Sönke Ludwig wrote:
> My favorite solution would be to introduce a simple #identifier.chain or
> @identifier.chain syntax to let the compiler or documentation generator
> insert the proper links or macro calls. Using a sepecial $(REF
> identifier.chain) pseudo macro, like Jacob suggested, would work as
> well, but I can't see how the additional syntax/visual/typing overhead
> is justified for such a common feature - at least when trying to find an
> optimal solution, compared to the current macro based system it would of
> course still be a big improvement.

I agree that a good compromise would be to lighten the syntactic overhead.

How about using the $(D) macro instead of $(REF)? This macro syntax-highlights its parameter as D code, and in Phobos it is already used for identifiers, so I think a good solution would be to find and create cross-reference links only for $(D contents).

Also, I'd like to mention a previous idea I posted here: this problem, and several other problems we have with DDoc, would be gone if we'd put what we already have with what we already have, and allow writing D code that gets interpreted to translate documentation comments into documentation output, using our existing CTFE capabilities. This way, the entire DDoc syntax can be reimplemented using "library" D code, interpreted by the compiler during documentation generation, and anyone could build upon it or completely replace it with another syntax.

P.S. How come your user agent (Thunderbird) is not emitting format=flowed messages? According to [1], it supports format=flowed, and can only be disabled via editing option strings.

[1]: http://kb.mozillazine.org/Plain_text_e-mail_(Thunderbird)#Flowed_format

Am Tue, 31 Dec 2013 10:28:46 +0100
schrieb Sönke Ludwig <sludwig@outerproduct.org>:

> Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:
> > On 12/30/13 1:00 PM, Walter Bright wrote:
> >> On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
> >>> Oh the default is to link and then disable manually? That's still
> >>> manual, just
> >>> the default is different :o). I can work with that.
> >>
> >> As I mentioned elsewhere, I dislike the requirement to use the inverse feature. Consider maintenance - I add a symbol, and then have to grep the rest of the module for any inadvertent collisions? Features should be positive ones, not negative.
> > 
> > Your arguments are convincing.
> > 
> > Andrei
> > 
> 
> The argument in isolation is sound, no doubt. But alone it doesn't lead to a solution*. There are a lot of arguments regarding the current situation and if the decision is not to use the existing _negative rule, then all of those should be carefully taken into account.
> 
>  - Visual noise and effort when using $(XREF) style macros
>  - $(XREF) isn't sufficient! A multitude of macros is actually needed
>  - There is no standard set of macros, it's project specific
>  -> The whole documentation needs to be revised when moving code between
>     projects!
>  - Macros need to be carefully crafted so that they don't encode
>    knowledge about the final documentation/file structure
>  - Macros alone can't cope with nested directory structures
>  - The cognitive load for multiple such macros would very likely cause
>    them to go unused in many cases
> 
> My favorite solution would be to introduce a simple #identifier.chain or @identifier.chain syntax to let the compiler or documentation generator insert the proper links or macro calls. Using a sepecial $(REF identifier.chain) pseudo macro, like Jacob suggested, would work as well, but I can't see how the additional syntax/visual/typing overhead is justified for such a common feature - at least when trying to find an optimal solution, compared to the current macro based system it would of course still be a big improvement.
> 
> But still, considering that there is precedence for the _negative approach (i.e. the problems of it are already there), I find using that acceptable, too, even if the concept is not as pretty/clean.
> 
> 
> * *Some* solution is needed. The different structure of the DDOX documentation for example makes the $(XREF) macros worthless, so they are currently completely disabled and automatic link generation is used instead.

I'm fully with you on that one and I would like to add that
DDoc's macro system reverses the responsibilities. Instead of
writing easy to read documentation that is agnostic to the
target output, you have to take into account any potential
output format's peculiarities.
This starts with the list of escaped characters in your
custom template and ends with structuring elements for lists
and tables. Without abstraction of structuring elements in
DDoc lists and tables look like HTML and are likely not
portable to other formats that use a different or implicit
structuring scheme. E.g. if you wanted to support MarkDown with
its explicit numbers for ordered lists, you'd have to replace
all $(LI) with $(LI1), $(LI2), ...
I guess it is a little too late to ask for a complete
reimplementation of DDoc?

-- 
Marco