(I don't know if this is the right forum for this post, so if not please let me know.)

I'm working on adding Markdown support to DDoc. I'm doing it by modifying the highlightText method in doc.d [1].

I've been trying to make my changes non-breaking, in that existing DDoc files should be rendered the same as they currently are. However I've realized that there may need to be some breakage, and I'd like to discuss it.

First of all, the CSS styles in the default DDoc theme remove all styling from headers, blockquotes, list items, pre and code sections, etc. [2] The effect of this is that Markdown features like headers, quotes, etc. don't display correctly under the default theme. I'd like to modify the default DDoc theme to avoid de-styling these elements.

Secondly, HTML is treated differently under Markdown and DDoc. In DDoc anything directly within an HTML tag (i.e. between < and >) is passed through unchanged. However if it's not part of the tag it is treated as regular DDoc text and subject to regular processing. In Markdown all text between a given set of HTML tags is passed through unchanged [3]. I'd prefer to switch to the CommonMark handling, allowing whole blocks of HTML to be passed through directly.

Thirdly and most controversially is how DDoc handles blank lines between paragraphs. Markdown wraps paragraphs in <p></p> tags and collapses multiple blank lines between paragraphs (i.e. ten blank lines is treated the same as one). DDoc inserts $(DDOC_BLANKLINE) macros on each blank line it encounters, which in the default theme inserts <br><br>, in the Phobos theme inserts <div class="blankline"/>, and for LaTeX inserts /par. Phobos docs also have the common practice of hand-wrapping paragraphs in $(P) macros which inserts <p></p> tags.

The result of DDoc's handling is that under the default theme if you have, say, three blank lines between paragraphs then it results in five blank lines being rendered between them.

I'm not yet sure it's feasible, but I'd really like to change DDoc to treat paragraphs the way Markdown does--wrap them in, say, a $(P) macro and collapse multiple blank lines. Any desired extra blank lines could be inserted by hand, via the $(BR) macro or similar. However this could change the rendering of existing documentation which depends on the $(DDOC_BLANKLINE) macro.

What do you think about the proposed changes? Should I try to implement any or all of them?

-Dave


Appendix A: Planned Markdown support
------------------------------------

I plan to support the following Markdown features, as specified by CommonMark [4]:

- Headings, both the ATX and setext kind
- Styling text with * (but not with _)
- Quoting text blocks with >
- Code blocks fenced with ```, ~~~ and DDoc's existing ---
- Backtick-quoted code as it currently works in DDoc
- Nesting lists
- Thematic breaks (<hr>)
- Inline links, although I'm not sure if I will support reference links [5]
- Markdown escapes with \
- HTML, although that's a discussion item (see above)

I don't plan to support the following Markdown features:

- Indented code blocks because existing DDoc comments are often already indented
- Using --- for HRs and second-level headings because DDoc uses it as a code block fence
- Tables
- Backtick-quoting code across multiple lines
- Github-specific features like task lists, mentioning people and teams, referencing issues and pull requests, and emoji


[1] https://github.com/dgileadi/dmd/tree/ddoc-markdown
[2] https://github.com/dlang/dmd/blob/master/res/default_ddoc_theme.ddoc#L70
[3] http://spec.commonmark.org/0.28/#html-blocks
[4] http://spec.commonmark.org/0.28/
[5] http://spec.commonmark.org/0.28/#reference-link

I think it's great that you're working on implementing it!

I don't have time at the moment to go over your proposal, but here's the previous long thread on it:

http://www.digitalmars.com/d/archives/digitalmars/D/Adding_Markdown_to_Ddoc_308861.html

Regarding breaking changes:

There's far too much existing Ddoc code. I don't think we can afford significant breaking changes.

On 1/20/18 1:45 PM, Walter Bright wrote:
> I think it's great that you're working on implementing it!
> 
> I don't have time at the moment to go over your proposal, but here's the previous long thread on it:
> 
> http://www.digitalmars.com/d/archives/digitalmars/D/Adding_Markdown_to_Ddoc_308861.html 
> 
> 
> Regarding breaking changes:
> 
> There's far too much existing Ddoc code. I don't think we can afford significant breaking changes.

I understand. When you get a few moments, I hope you can take a look at what I'm proposing.

On 2018-01-20 19:33, David Gileadi wrote:

> First of all, the CSS styles in the default DDoc theme remove all styling from headers, blockquotes, list items, pre and code sections, etc. [2] The effect of this is that Markdown features like headers, quotes, etc. don't display correctly under the default theme. I'd like to modify the default DDoc theme to avoid de-styling these elements.

I'm the author of the current Ddoc default theme. As far as I know it's common practice to reset the style for standard tags because the browsers might render it differently otherwise. If any tags are not render the way you like, just add the appropriate styling. But please, do not remove the resetting of the style for the standard tags.

-- 
/Jacob Carlborg

On 1/21/18 10:55 AM, Jacob Carlborg wrote:
> On 2018-01-20 19:33, David Gileadi wrote:
> 
>> First of all, the CSS styles in the default DDoc theme remove all styling from headers, blockquotes, list items, pre and code sections, etc. [2] The effect of this is that Markdown features like headers, quotes, etc. don't display correctly under the default theme. I'd like to modify the default DDoc theme to avoid de-styling these elements.
> 
> I'm the author of the current Ddoc default theme. As far as I know it's common practice to reset the style for standard tags because the browsers might render it differently otherwise. If any tags are not render the way you like, just add the appropriate styling. But please, do not remove the resetting of the style for the standard tags.

There are different opinions on the amount of resets; some CSS resets preserve some styles while normalizing others. However I don't have a strong opinion either way so I'll stick with your recommendation and add back the necessary styles. Once (if?) this makes it to a PR I'd appreciate your input on the chosen styles, to help them to better fit the theme.

On 1/20/2018 10:33 AM, David Gileadi wrote:
> I've been trying to make my changes non-breaking, in that existing DDoc files should be rendered the same as they currently are. However I've realized that there may need to be some breakage, and I'd like to discuss it.
> 
> First of all, the CSS styles in the default DDoc theme remove all styling from headers, blockquotes, list items, pre and code sections, etc. [2] The effect of this is that Markdown features like headers, quotes, etc. don't display correctly under the default theme. I'd like to modify the default DDoc theme to avoid de-styling these elements.

I don't understand how styling enters into this. Markdown, such as:

   * cat
   * dog
   * horse

should be simply rewritten by the highlighter to:

   $(UL
   $(LI cat)
   $(LI dog)
   $(LI horse)
   )

and then let the macros do their thing.


> Secondly, HTML is treated differently under Markdown and DDoc. In DDoc anything directly within an HTML tag (i.e. between < and >) is passed through unchanged. However if it's not part of the tag it is treated as regular DDoc text and subject to regular processing. In Markdown all text between a given set of HTML tags is passed through unchanged [3]. I'd prefer to switch to the CommonMark handling, allowing whole blocks of HTML to be passed through directly.

When Markdown and Ddoc conflict, do it the Ddoc way.


> Thirdly and most controversially is how DDoc handles blank lines between paragraphs. Markdown wraps paragraphs in <p></p> tags and collapses multiple blank lines between paragraphs (i.e. ten blank lines is treated the same as one). DDoc inserts $(DDOC_BLANKLINE) macros on each blank line it encounters, which in the default theme inserts <br><br>, in the Phobos theme inserts <div class="blankline"/>, and for LaTeX inserts /par. Phobos docs also have the common practice of hand-wrapping paragraphs in $(P) macros which inserts <p></p> tags.
> 
> The result of DDoc's handling is that under the default theme if you have, say, three blank lines between paragraphs then it results in five blank lines being rendered between them.
> 
> I'm not yet sure it's feasible, but I'd really like to change DDoc to treat paragraphs the way Markdown does--wrap them in, say, a $(P) macro and collapse multiple blank lines. Any desired extra blank lines could be inserted by hand, via the $(BR) macro or similar. However this could change the rendering of existing documentation which depends on the $(DDOC_BLANKLINE) macro.
> 
> What do you think about the proposed changes? Should I try to implement any or all of them?

There is so much written in Ddoc these days that fundamentally changing how it works would be very upsetting to a lot of people who have their documents broken, for very little benefit.

What I mean by this is there is no existing Markdown text that will be imported into Ddoc, so this does not have to be accommodated. Supporting some Markdown features is for convenience for Ddoc users. Where Ddoc and Markdown conflict, Ddoc takes precedence because there is a lot of legacy Ddoc text and zero legacy Markdown text.



> Appendix A: Planned Markdown support
> ------------------------------------
> 
> I plan to support the following Markdown features, as specified by CommonMark [4]:
> 
> - Headings, both the ATX and setext kind

No reason to support both. ATX looks simpler to me, so go with that.

> - Styling text with * (but not with _)

Yes, just stick with one. The * is good.

> - Quoting text blocks with >

Ok.

> - Code blocks fenced with ```, ~~~ and DDoc's existing ---

The existing --- is good, no need to add more options.

> - Backtick-quoted code as it currently works in DDoc

Ok. (This has been a big success in Ddoc.)

> - Nesting lists

   1. cat
   2. dog
   3. box

is fine for ordered lists, using * for unordered ones is plenty good enough.

> - Thematic breaks (<hr>)

Just pick one style for that, I suggest the 3 underscores.

> - Inline links, although I'm not sure if I will support reference links [5]

Ddoc already recognizes URLs and creates links for them

> - Markdown escapes with \

Ok

> - HTML, although that's a discussion item (see above)

I wouldn't change the HTML behavior.

> I don't plan to support the following Markdown features:
> 
> - Indented code blocks because existing DDoc comments are often already indented
> - Using --- for HRs and second-level headings because DDoc uses it as a code block fence
> - Tables
> - Backtick-quoting code across multiple lines
> - Github-specific features like task lists, mentioning people and teams, referencing issues and pull requests, and emoji

Yes, don't waste time on those things.

And thank you for doing this! This will be a nice addition to Ddoc.

On Monday, 22 January 2018 at 05:03:41 UTC, Walter Bright wrote:
> On 1/20/2018 10:33 AM, David Gileadi wrote:
> {snip}
>
>> Appendix A: Planned Markdown support
>> ------------------------------------
>> 
>> I plan to support the following Markdown features, as specified by CommonMark [4]:
>> 
>> {snip}
>
>> - Nesting lists
>
>    1. cat
>    2. dog
>    3. box
>
> is fine for ordered lists, using * for unordered ones is plenty good enough.

For maximum ease of reading, nesting, and for consistency, I suggest following the 4-space rule. That is, list item *content* (not the markers) always starts at multiples of 4 columns in; additional paragraphs of a list item are also indented 4 spaces; code blocks indented 4 spaces. For example: <https://gist.github.com/uvtc/379bca988270ba8b44a30efabbfa9d6b>.

Thanks for looking this over.

On 1/21/18 10:03 PM, Walter Bright wrote:
> On 1/20/2018 10:33 AM, David Gileadi wrote:
>> I've been trying to make my changes non-breaking, in that existing DDoc files should be rendered the same as they currently are. However I've realized that there may need to be some breakage, and I'd like to discuss it.
>>
>> First of all, the CSS styles in the default DDoc theme remove all styling from headers, blockquotes, list items, pre and code sections, etc. [2] The effect of this is that Markdown features like headers, quotes, etc. don't display correctly under the default theme. I'd like to modify the default DDoc theme to avoid de-styling these elements.
> 
> I don't understand how styling enters into this. Markdown, such as:
> 
>     * cat
>     * dog
>     * horse
> 
> should be simply rewritten by the highlighter to:
> 
>     $(UL
>     $(LI cat)
>     $(LI dog)
>     $(LI horse)
>     )
> 
> and then let the macros do their thing.

Right, and that's exactly what I'm doing. Styling enters into it because Ddoc ships with a default theme, and that theme doesn't render things like headings properly. With Markdown support Ddoc will now generate headings, so the default theme needs to support them too. I'll follow Jacob's suggestion for adding styles for headings and other elements.

I'll leave HTML and blank line handling as-is.

[snip]

> 
>> Appendix A: Planned Markdown support
>> ------------------------------------
>>
>> I plan to support the following Markdown features, as specified by CommonMark [4]:
>>
>> - Headings, both the ATX and setext kind
> 
> No reason to support both. ATX looks simpler to me, so go with that.
> 
>> - Styling text with * (but not with _)
> 
> Yes, just stick with one. The * is good.
> 
>> - Quoting text blocks with >
> 
> Ok.
> 
>> - Code blocks fenced with ```, ~~~ and DDoc's existing ---
> 
> The existing --- is good, no need to add more options.
> 
>> - Backtick-quoted code as it currently works in DDoc
> 
> Ok. (This has been a big success in Ddoc.)
> 
>> - Nesting lists
> 
>     1. cat
>     2. dog
>     3. box
> 
> is fine for ordered lists, using * for unordered ones is plenty good enough.
> 
>> - Thematic breaks (<hr>)
> 
> Just pick one style for that, I suggest the 3 underscores.
> 
>> - Inline links, although I'm not sure if I will support reference links [5]
> 
> Ddoc already recognizes URLs and creates links for them
> 
>> - Markdown escapes with \
> 
> Ok
> 
>> - HTML, although that's a discussion item (see above)
> 
> I wouldn't change the HTML behavior.
> 
>> I don't plan to support the following Markdown features:
>>
>> - Indented code blocks because existing DDoc comments are often already indented
>> - Using --- for HRs and second-level headings because DDoc uses it as a code block fence
>> - Tables
>> - Backtick-quoting code across multiple lines
>> - Github-specific features like task lists, mentioning people and teams, referencing issues and pull requests, and emoji
> 
> Yes, don't waste time on those things.
> 
> And thank you for doing this! This will be a nice addition to Ddoc.

You're very welcome.

Regarding multiple ways of doing headings, thematic breaks, unordered lists, etc. I'd really like to support as much standard Markdown as possible. My rationale is:

1. It's easier for people who already know Markdown to use when the list of unsupported features is small.
2. There is already harboured-mod which melds Ddoc and Markdown, and I've been trying to match its list of differences from vanilla Markdown [1] so that it's straightforward to switch between DDoc generators.
3. The additional code for implementing the same feature with different triggering characters is small.

Incidentally, to me help stick to the CommonMark spec I'm using its test suite to generate a Ddoc page that tests the Markdown support [2]. I'm currently at about 40% compliance. When I'm done I plan to make the generated page part of DDoc's test suite.


[1] https://github.com/dlang-community/harbored-mod#differences-from-vanilla-markdown
[2] https://github.com/dgileadi/ddoc-commonmark-spec

On 1/22/18 10:37 AM, John Gabriele wrote:
> On Monday, 22 January 2018 at 05:03:41 UTC, Walter Bright wrote:
>> On 1/20/2018 10:33 AM, David Gileadi wrote:
>> {snip}
>>
>>> Appendix A: Planned Markdown support
>>> ------------------------------------
>>>
>>> I plan to support the following Markdown features, as specified by CommonMark [4]:
>>>
>>> {snip}
>>
>>> - Nesting lists
>>
>>    1. cat
>>    2. dog
>>    3. box
>>
>> is fine for ordered lists, using * for unordered ones is plenty good enough.
> 
> For maximum ease of reading, nesting, and for consistency, I suggest following the 4-space rule. That is, list item *content* (not the markers) always starts at multiples of 4 columns in; additional paragraphs of a list item are also indented 4 spaces; code blocks indented 4 spaces. For example: <https://gist.github.com/uvtc/379bca988270ba8b44a30efabbfa9d6b>.

Thanks for the suggestion. However, since I'm trying to stick to the CommonMark spec I've implemented their rules for determining when indented text continues a list item, etc.

I'm currently not supporting indented code blocks, however. I could potentially support them within list items, but right now I don't intend to.

On Monday, 22 January 2018 at 18:17:37 UTC, David Gileadi wrote:
> On 1/22/18 10:37 AM, John Gabriele wrote:
>> On Monday, 22 January 2018 at 05:03:41 UTC, Walter Bright wrote:
>>> On 1/20/2018 10:33 AM, David Gileadi wrote:
>>> {snip}
>>>
>>>> Appendix A: Planned Markdown support
>>>> ------------------------------------
>>>>
>>>> I plan to support the following Markdown features, as specified by CommonMark [4]:
>>>>
>>>> {snip}
>>>
>>>> - Nesting lists
>>>
>>>    1. cat
>>>    2. dog
>>>    3. box
>>>
>>> is fine for ordered lists, using * for unordered ones is plenty good enough.
>> 
>> For maximum ease of reading, nesting, and for consistency, I suggest following the 4-space rule. That is, list item *content* (not the markers) always starts at multiples of 4 columns in; additional paragraphs of a list item are also indented 4 spaces; code blocks indented 4 spaces. For example: <https://gist.github.com/uvtc/379bca988270ba8b44a30efabbfa9d6b>.
>
> Thanks for the suggestion. However, since I'm trying to stick to the CommonMark spec I've implemented their rules for determining when indented text continues a list item, etc.

AFAICT, CommonMark does indeed follow the 4-space rule, though may also provide the extra flexibility of leaving out some prefixing spaces. Try copying and pasting my gist example (above) into [the CommonMark dingus](http://spec.commonmark.org/dingus/). It works as expected.

Note also, for multiple lines in a list item, the continued lines line up vertically with the first (same indentation throughout). It's really a nifty consistency across all list types that makes nesting clear cut.