On Sunday, 26 May 2013 at 01:57:16 UTC, Andrei Alexandrescu wrote:
> This is a worthy goal. We manage to generate mobi files for the spec (and Phobos in a pull request), is the ebook format very different?
>
> Andrei

Good evening, Professor,

I'm still working through the ePUB standard, but, from what I can tell, the two are very similar. According to Wikipedia, ePub obsoletes Mobi (despite Amazon stubbornly enforcing Mobi (or some variant) for its Kindle). They both rely on similar standards, including OPF and NCX file formats, and ePUB uses XHTML and zip compression, thereby avoiding reinventing the wheel to the extent possible (although one would think that Docbook would make more sense for, you know, eBooks).

Actually, in generating my first ePUB of the DLang spec, I simply used the same mobi input files, copied-and-pasted two boilerplate ePUB files and zipped the whole thing up. Aside from some egregious formatting on my Kobo, it works. Of course, I want to take the time to generate a proper ePUB 3 file and use as much of the standard as possible to advantage.

Naturally, the goal is to extend whatever scripts I write to generate a DLang spec ePub to generate DDoc documentation eBooks, too, so I'm not entirely a one-issue person...

On 5/25/2013 5:28 PM, Andrei Alexandrescu wrote:
> My attitude on DDoc has evolved in threes:
>
> 3 minutes: "wtf is this crap"
> 3 hours: "this sucks"
> 3 days: "grumble I'll make do with this although it totally sucks"
> 3 months: "this is pretty darn good"

Thanks for the chuckle!


> DDoc is simple for what it does, it has somehow hit a sweet spot.

It's not totally random. I've designed one macro language before (ABEL), and have implemented 3 (ABEL, Make, and C preprocessor), so I knew what I wanted. Ddoc is very similar to Make's macro system.

BTW, the C preprocessor takes the cake for being both horrendously complicated (most implementations take about 10 years to get right) and woefully inadequate.

On Sunday, May 26, 2013 02:44:30 Borden wrote:
> On Saturday, 25 May 2013 at 23:28:46 UTC, Jonathan M Davis wrote:
> > aside from you, I'm not aware of anyone complaining about it any time recently.
> 
> Good evening, Jonathan,
> 
> I'm not sure whether you mean that nobody's complained recently about the spec being in DDoc lately, because, as in my first link, I found that more people disliked the macros feature of DDoc (2 + me) than liked it (0).

AFAIK, your recent posts on ddoc are the first that anyone has complained about it in quite some time. There are plenty of folks who want various improvements to the online documentation, but doesn't necessarily require doing anything to ddoc, and it's rarely the case that someone complains about ddoc itself.

> To answer your question, regarding concrete examples, these issues are in the context of wanting to translate the DLangSpec pages into HTML5 so that I can compile them into an ePUB document:
> 
> 1) the DLangSpec files (pick any one) use SECTION# macros, where # is a number. In the DDoc conversion files to HTML, these SECTION# macros convert to <h#> tags and encase the contents within them. However, I'm not aware of any facility within DDoc, short of hand-writing a parser, to allow these SECTION# macros to be nested in order to take advantage of HTML5's <section> tags.

Normally, you'd nest things by nesting macros. e.g.

$(NESTED stuff $(NESTED more stuff $(NESTED yet more stuff) $(NESTED other stuff)))

But I'm afrad that my understanding of html (and particularly html 5) is limited enough that you would have to give explict code samples for me to see what you can't convert.

However, ddoc should allow you to do pretty much anything that involves simply transforming the content of a macro to somethnig else. The macro takes a set of arguments and then creates something new with them by rearranging them and adding stuff around them and the like. So,

$(MACRO foo, bar, fiddly)

can become

<foo><fiddly>bar</fiddly></foo>

or

foo!bar.fiddly

or whatever other combination of textual replacement and reording that you want to do. It's things that require counting what's there or generating something elsewhere in tho document base on macros (such as a table of contents or an index) which it can't do on its own (though it would be trivial to have another program read the ddoc and manipulate it to create sections for table of contents and the like if want to do that with a document that you're writing with ddoc).

The exact set of macros used with the online documentation may very well be too specific to html 4, and it may be that the macros will have to be rewritten and moved around in the documentation, but the macro system itself will almost certainly do what you want.

> 2) The macros are not self-documenting. For example, consider $(LNAME2 pointers, Pointers) in arrays.dd. The easiest way, I know, to figure out what $(LNAME2) means is to read the posix.mak to see that arrays.dd gets pumped through ddoc.dd. Now, a search through doc.ddoc to find the declaration LINK2=<a href="$1">$+</a> at last tells me that argument 1 is the path to the link and everything that follows that is the text to appear in the link. The point is that, as I struggle through modifying the existing .ddoc templates to compile to HTML5, I need to keep flipping back and forth between the source and the .ddoc to make sure that anything I'm redefining I'm doing correctly.

The same goes for any function name. You frequently have to look them up to see exactly what they do. If they have better names, that helps, but how self- documenting a macro is is completely up to how well it was named. That has nothing to do with ddoc itself.

> 3) Again using LINK2, if I were to delete the LINK2= line from doc.ddoc and forget to readd it, my experience is that dmd -D will quietly drop instances of $(LINK2) without telling me.

Then perhaps dmd should be fixed so that it complains. That's a quality of implementation issue and probably easily fixed.

> 4) Again using the same example, if LINK2 gets defined in multiple DDoc files, how do I know for certain which definition it calls when dmd runs against the files?

Again. That's a QoI issue. We can probably make the compiler give a warning or error in that case.

> 5) I find that a lot of the DLangSpec is written from an HTML point of view, so maybe it just needs rewriting to make the macros descriptive. For example, consider $(B dmd) and $(B -gc) on lines 881-882 of abi.dd. By default, these get converted into <b>dmd</b> and <b>-gc</b> Say I want commands (like dmd) to be bolded but I want command-line arguments not to be bolded. There's no way to write B= to single out some $(B)s and not others. Now, I know the knee-jerk response is "Yes, but HTML works the exact same way." That's true, but CSS *does* give you a bunch of selectors to cherry-pick, say, "only the <b> tags of class X" or "the element with this id." Meaning that all I have to do is find the Bs I want to change and add a class="" without having to worry about updating any of the other Bs. Is there a simple way to do this in DDoc?

Ddoc is pure macros. If you want different stuff to be treated differently, you need different macros. But css can be used just fine with the html generated by ddoc, so if css supports what you're looking for, then I would think that you'd be able to do it with css.

Regardless, the exact set of macros used with dlang.org are definitely targeted at html, so it's quite possible that we'll need a different set of macros to get it to convert to other formats more easily (though it already translates to one ebook format as well as latex - though Andrei is working on improving the latex stuff). So, my guess is that this not a problem with ddoc but rather a problem with how ddoc is being used.

- Jonathan M Davis

Good evening, Professor,

On Sunday, 26 May 2013 at 02:05:55 UTC, Andrei Alexandrescu wrote:
> What vexes me is that all the sugar you propose goes against what you opened with...

I'm not trying to cause any offence, and I apologise if any of my phrasing or comments are construed that way. I know that I can be a little bit terse at times but I hope it's taken in the best possible way because I have full respect for the design and implementation of the language.

On Sunday, 26 May 2013 at 02:05:55 UTC, Andrei Alexandrescu wrote:
> I can hypothesize that the shortest  path between where we are
> and what you're trying to accomplish is a few dozens of macro
> definitions. Did you try doing that and failed?

Indeed, it has run into some snags:

1) One of the first problems I ran into was coming up with rules for spec.dd. My original objective was to enclose the $(TOC) macro into <nav> tags, consistent both with the HTML5 spec and ePUB3. However, <p> tags are not allowed within the <nav> tags, but I also don't want to strip out the explanation the informative information. I don't know how to define TOC to keep the $(P) macro outside of the <nav> element which will enclose the TOCENTRY items.

2) Consider, for example, parsing arrays.dd (my comment can be easily applied to any other file). Unless I'm miscounting parentheses, $(H4) macros are not being used within $(H3) macros. Therefore, how do I get DDoc to parse the file so that it ends up with nested <section> tags? for example:
<section><h3>Dynamic Arrays</h3>
<section><h4>Array Declarations</h4>
Content
</section></section>

3) Consider, for example, this part from abi.dd:
$(GRAMMAR
$(I MangledName):
    $(B _D) $(I QualifiedName) $(I Type)
    $(B _D) $(I QualifiedName) $(B M) $(I Type)

$(I QualifiedName):
    $(I SymbolName)
    $(I SymbolName) $(I QualifiedName)

$(I SymbolName):
    $(I LName)
    $(I TemplateInstanceName)
)
Say I want to style this using a descriptions list, the <dl> tag. That's easy enough, but now how do I tell DDoc to tag the $(I) macros using <dt> and <dd> tags?

4) Furthermore (still referring to the example above, because the issue applies to other areas), how do I tell DDoc that $(I)s within a $(GRAMMAR) macro are to be formatted using descriptions list syntax, but keep the other $(I) macros as regular <i> elements?

5) The link-related macros appear, by and large, to use relative URLs. If I'm compiling only the DLang Spec into an ePUB, the standard, I believe, requires that the links be resolvable. That's easy enough if the relative URL in question points to another page in the spec. However, if the link points to another page on the website or a library document, which (for now) won't be in the ePUB, is the only way to identify and fix those links going to be by hand?

These are the problems that I've run into thus far. I'm doing my best to appreciate the design and theory of DDoc, but maybe it's too much of a paradigm shift for me and I end up fighting the macros?

On Sunday, 26 May 2013 at 03:51:48 UTC, Walter Bright wrote:
> It's not totally random. I've designed one macro language before (ABEL), and have implemented 3 (ABEL, Make, and C preprocessor), so I knew what I wanted. Ddoc is very similar to Make's macro system.
>
> BTW, the C preprocessor takes the cake for being both horrendously complicated (most implementations take about 10 years to get right) and woefully inadequate.

Good evening, Walter,

I noticed the similarities to DDoc macros and Make immediately. Again, I think the documentation system you designed is excellent. The 'sugar' I suggested in an earlier post seemed, at least to me, in line with the general 'common sense syntax' that you implemented elsewhere - such as with defining code, variables, dates, authors, paragraphs, etc. Then again, you're dealing with an (aspiring) accountant, not a computer scientist, so I only have experience in trying to make complex things look pretty and not caring about all of that optimisation and implementation stuff!

Again, I have nothing but the highest respect for the work you've done. Think of it this way: if I didn't admire D so much, I wouldn't be so determined to get its documentation onto eReaders!

On 5/25/2013 9:15 PM, Borden wrote:
> 3) Consider, for example, this part from abi.dd:
> $(GRAMMAR
> $(I MangledName):
>      $(B _D) $(I QualifiedName) $(I Type)
>      $(B _D) $(I QualifiedName) $(B M) $(I Type)
>
> $(I QualifiedName):
>      $(I SymbolName)
>      $(I SymbolName) $(I QualifiedName)
>
> $(I SymbolName):
>      $(I LName)
>      $(I TemplateInstanceName)
> )
> Say I want to style this using a descriptions list, the <dl> tag. That's easy
> enough, but now how do I tell DDoc to tag the $(I) macros using <dt> and <dd> tags?

You wouldn't use the I macro. You'd write a macro that reflected the structure - you might want to look at the TABLE, TROW, THEAD, THX and TDX macros to see how tables are generated in a flexible, structured manner.

On 5/25/2013 8:55 PM, Jonathan M Davis wrote:
>> 3) Again using LINK2, if I were to delete the LINK2= line from
>> doc.ddoc and forget to readd it, my experience is that dmd -D
>> will quietly drop instances of $(LINK2) without telling me.
>
> Then perhaps dmd should be fixed so that it complains. That's a quality of
> implementation issue and probably easily fixed.

It's quite deliberate, is not a QoI issue, and doesn't need to be fixed.


>> 4) Again using the same example, if LINK2 gets defined in
>> multiple DDoc files, how do I know for certain which definition
>> it calls when dmd runs against the files?
>
> Again. That's a QoI issue. We can probably make the compiler give a warning or
> error in that case.

Again, this is deliberate. Macros are set up so that the last one overrides all the previous ones, enabling a hierarchy of them using ddoc files. It's a simple form of 'inheritance'.

On Sunday, 26 May 2013 at 03:56:08 UTC, Jonathan M Davis wrote:
> AFAIK, your recent posts on ddoc are the first that anyone has complained about it in quite some time. There are plenty of
> folks who want various improvements to the online documentation,
> but doesn't necessarily require doing anything to ddoc, and it's
> rarely the case that someone complains about ddoc itself.

That's fair, and has probably only come up now because I've decided - granted with very little experience in DDoc - to kick the proverbial hornets' nest by diving head-first into it and do crazy things with the source.

On Sunday, 26 May 2013 at 03:56:08 UTC, Jonathan M Davis wrote:
> Normally, you'd nest things by nesting macros. e.g.
>
> $(NESTED stuff $(NESTED more stuff $(NESTED yet more stuff) $(NESTED other
> stuff)))

Indeed. I suppose, in addition to my grievances about error checking, is suggesting that DDoc should include parentheses closure checking?

On Sunday, 26 May 2013 at 03:56:08 UTC, Jonathan M Davis wrote:
> However, ddoc should allow you to do pretty much anything that involves simply transforming the content of a macro to somethnig else. The macro takes a set of arguments and then creates something new with them by rearranging them and adding stuff around them and the like.

And maybe that's what my biggest frustration with the macros is (or at least how they're implemented in the DLang spec): they read like an abstracted wrapper for HTML, and someone like me immediately yearns for the extra features that got simplified out.

Say, for example, we have a $(B) macro, and I want some of them to have ids or classes and others not (since my eventual CSS file will have special formatting rules for them). To add this functionality, I would have to find all of the $(B)s and rewrite them to say $(B id, class, content). For each one that I miss, I'm going to have an empty <b> element with its id set to the content!

On Sunday, 26 May 2013 at 03:56:08 UTC, Jonathan M Davis wrote:
> The exact set of macros used with the online documentation may very well be too specific to html 4, and it may be that the
> macros will have to be rewritten and moved around in the
> documentation

And now I think we're getting to the heart of the problem. I might have a more favourable opinion of macros if they were more descriptive of the content in the DLang spec source files. That would allow a fair bit more flexibility.

Still, I don't think there's any avoiding that any macros requiring nested formatting or special parameters will require them to be written with their intended output formats 'in mind' to work correctly. And, of course, I am very reluctant to attack any of this lest I start breaking the website, Latex or PDF generation in getting the DLang spec 'HTML5 ready'.

Hence, why I'm at a standstill.

On Sunday, 26 May 2013 at 04:30:46 UTC, Walter Bright wrote:
> Again, this is deliberate. Macros are set up so that the last one overrides all the previous ones, enabling a hierarchy of them using ddoc files. It's a simple form of 'inheritance'.

And perhaps this point could be clarified (and, when I next attack the source I'll test it). I have one.ddoc two.ddoc and src.dd. In src.dd, I use $(MY_MACRO x). one.ddoc has the line MY_MACRO=<p>Called one on $1</p>; two.ddoc has the line MY_MACRO=<p>Called two on $1</p>.

So, I now run dmd -o- -D one.ddoc two.ddoc src.dd. What does src.html say?

On Sunday, 26 May 2013 at 04:57:12 UTC, Borden wrote:
> On Sunday, 26 May 2013 at 04:30:46 UTC, Walter Bright wrote:
>> Again, this is deliberate. Macros are set up so that the last one overrides all the previous ones, enabling a hierarchy of them using ddoc files. It's a simple form of 'inheritance'.
>
> And perhaps this point could be clarified (and, when I next attack the source I'll test it). I have one.ddoc two.ddoc and src.dd. In src.dd, I use $(MY_MACRO x). one.ddoc has the line MY_MACRO=<p>Called one on $1</p>; two.ddoc has the line MY_MACRO=<p>Called two on $1</p>.
>
> So, I now run dmd -o- -D one.ddoc two.ddoc src.dd. What does src.html say?

and by $1 I mean, of course, $0.