So, I've spent the better part of this morning tracing what seems to be a subtle bug in ddox' macro treatment. I defined these macros:

IGNORESECOND = [$1]
DOLLARZERO = dzbegin $0 dzend
TEST = before $(IGNORESECOND $(DOLLARZERO one, two)) after

I inserted $(TEST) in a Phobos module. When processing the module with ddoc, the macro expands to:

before [dzbegin one, two dzend] after

wherease with ddox it expands to:

before [dzbegin one] after

i.e. the comma "escapes" the confines of the macro DOLLARZERO.

The "right" behavior is ddoc's, for several reasons. Is there a distinct macro engine powering ddox generation? Who is maintaining that?


Thanks,

Andrei

On Friday, 16 September 2016 at 18:40:03 UTC, Andrei Alexandrescu wrote:
> So, I've spent the better part of this morning tracing what seems to be a subtle bug in ddox' macro treatment. I defined these macros:
>
> IGNORESECOND = [$1]
> DOLLARZERO = dzbegin $0 dzend
> TEST = before $(IGNORESECOND $(DOLLARZERO one, two)) after
>
> I inserted $(TEST) in a Phobos module. When processing the module with ddoc, the macro expands to:
>
> before [dzbegin one, two dzend] after
>
> wherease with ddox it expands to:
>
> before [dzbegin one] after
>
> i.e. the comma "escapes" the confines of the macro DOLLARZERO.
>
> The "right" behavior is ddoc's, for several reasons. Is there a distinct macro engine powering ddox generation? Who is maintaining that?
>
>
> Thanks,
>
> Andrei

I belive ddox is maintained by soenke ludwig

On 09/16/2016 03:11 PM, Stefan Koch wrote:
> I belive ddox is maintained by soenke ludwig

Thx, he's cc'd. Workaround for the time being: https://github.com/dlang/dlang.org/pull/1475, could someone please review. Thanks!

Andrei

There is indeed a separate macro engine [1] contained in the ddox repo. Neither DDMD, nor libddoc existed at the time and DMD's implementation was written in, from a D POV, very alien C style, with no suitable structure to extract high level information, so instead of porting it, I instead tried to implement a processor from the spec [1]. However, it turned out that the spec is not sufficient at all, so a lot of changes were necessary over time to work out those small differences. I'd like to improve the docs, but I'm still not confident that my implementation's behavior is always equivalent (as this issue shows).

I've opened a GitHub ticket now [2] and will have a look at it shortly.

[1]: https://github.com/rejectedsoftware/ddox/blob/master/source/ddox/ddoc.d
[2]: http://dlang.org/spec/ddoc.html#phases_of_processing
[3]: https://github.com/rejectedsoftware/ddox/issues/130

On 9/16/2016 11:53 PM, Sönke Ludwig wrote:
> There is indeed a separate macro engine [1] contained in the ddox repo. Neither
> DDMD, nor libddoc existed at the time and DMD's implementation was written in,
> from a D POV, very alien C style, with no suitable structure to extract high
> level information, so instead of porting it, I instead tried to implement a
> processor from the spec [1]. However, it turned out that the spec is not
> sufficient at all, so a lot of changes were necessary over time to work out
> those small differences. I'd like to improve the docs, but I'm still not
> confident that my implementation's behavior is always equivalent (as this issue
> shows).

I've factored out the Ddoc macro code into a separate component:

https://digitalmars.com/sargon/textmac.html

On 9/17/16 2:53 AM, Sönke Ludwig wrote:
> There is indeed a separate macro engine [1] contained in the ddox repo.
> Neither DDMD, nor libddoc existed at the time and DMD's implementation
> was written in, from a D POV, very alien C style, with no suitable
> structure to extract high level information, so instead of porting it, I
> instead tried to implement a processor from the spec [1]. However, it
> turned out that the spec is not sufficient at all, so a lot of changes
> were necessary over time to work out those small differences. I'd like
> to improve the docs, but I'm still not confident that my
> implementation's behavior is always equivalent (as this issue shows).
>
> I've opened a GitHub ticket now [2] and will have a look at it shortly.
>
> [1]:
> https://github.com/rejectedsoftware/ddox/blob/master/source/ddox/ddoc.d
> [2]: http://dlang.org/spec/ddoc.html#phases_of_processing
> [3]: https://github.com/rejectedsoftware/ddox/issues/130

Cool beans, thanks. Do you think we should unify the implementations? Perhaps by moving your code (assuming we get semantic convergence) into dmd so we get the benefit of hyphenation? Say e.g. we want to add markdown input, should we continue with two codebases or unify? Thanks! -- Andrei

On 9/17/16 5:29 AM, Walter Bright wrote:
> On 9/16/2016 11:53 PM, Sönke Ludwig wrote:
>> There is indeed a separate macro engine [1] contained in the ddox
>> repo. Neither
>> DDMD, nor libddoc existed at the time and DMD's implementation was
>> written in,
>> from a D POV, very alien C style, with no suitable structure to
>> extract high
>> level information, so instead of porting it, I instead tried to
>> implement a
>> processor from the spec [1]. However, it turned out that the spec is not
>> sufficient at all, so a lot of changes were necessary over time to
>> work out
>> those small differences. I'd like to improve the docs, but I'm still not
>> confident that my implementation's behavior is always equivalent (as
>> this issue
>> shows).
>
> I've factored out the Ddoc macro code into a separate component:
>
> https://digitalmars.com/sargon/textmac.html

Cool. The way the whole establishment should work is that all surface syntax rewrites itself into macros, and expand() is invoked as the last step. -- Andrei