I'm (finally) updating my dpldocs.info website again, and
before I just pulled names from the generated Phobos html.

This time, I want to use dmd's json output to avoid dependency
on the specific html layout of std.ddoc.


Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
one big .json file with all the info.

But, the comments aren't parsed at all.



So, my question is: do we have a ddoc parser that stands alone?


BTW: this is the complete listing from json so far:
http://dpldocs.info/search/awesome

I'm actually impressed with the speed, considering it is re-parsing
3 MB of generated json per load. std.variant and std.json might not
be speed demons, but they are plenty good enough for me.

The ultimate goal here is to revitalize my D search engine and to
make it as a nice index for all functions, with brief descriptions,
etc., hopefully for all D libs everywhere.

With the html, dealing with different people's macros to parse it
is a pain. But with the json, dmd always does it the same way, so
with some luck, I can just add a box "upload your project's json
file" and allow easy instant submissions for other libs.

On Wed, Aug 1, 2012 at 4:07 PM, Adam D. Ruppe <destructionator@gmail.com> wrote:
> I'm (finally) updating my dpldocs.info website again, and


Yes!

> Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got one big .json file with all the info.
>
> But, the comments aren't parsed at all.

Well, the comments are discarded as soon as lexical analysis is done, I guess. I suppose the DMD part that does the json output never ever sees any comment, just code tokens (see the discussion on lexing D)? I wonder how documentation generation is done with -D?

Since with the -D flag, DMD correctly 'attach' doc comments on code part, (this is a modul-level doc, this is a field documentation, ...) I guess the problem is to somehow transport this information to the json generator. This generator does not give much information btw, but my impression is that people do not use it that much and, consequently, no feature request is emitted.

On Wednesday, 1 August 2012 at 14:36:26 UTC, Philippe Sigaud wrote:
> Well, the comments are discarded as soon as lexical analysis is done, I guess.

They are actually there if you do both -X and -D together.
(-X alone doesn't show it; it must be discarded then, but
-D keeps it).


The problem is though that it is the text from the comment
without any organization or macro expansion.

For example:

{
"name" : "map(fun...) if (fun.length >= 1)",
"kind" : "template",
"protection" : "public",
"comment" : "\nImplements the homonym function (also known as $(D transform)) present\nin many languages of    functional flavor. The call $(D map!(fun)(range))\nreturns a range of which elements are obtained by applying  $(D fun(x))\nleft to right for all $(D x) in $(D range). The original ranges are\nnot changed. Evaluation is   done lazily. The range returned by $(D map)\ncaches the last value such that evaluating $(D front) multiple    times\ndoes not result in multiple calls to $(D fun).\n\nExample:\n----\nint[] arr1 = [ 1, 2, 3, 4 ];\nint[]   arr2 = [ 5, 6 ];\nauto squares = map!(\"a * a\")(chain(arr1, arr2));\nassert(equal(squares, [ 1, 4, 9, 16, 25, 36 ]));\n----\n\nMultiple functions can be passed to $(D map). In that case, the\nelement type of $(D map) is  a tuple containing one element for each\nfunction.\n\nExample:\n\n----\nauto arr1 = [ 1, 2, 3, 4 ];\nforeach   (e; map!(\"a + a\", \"a * a\")(arr1))\n{\n    writeln(e[0], \" \", e[1]);\n}\n----\n\nYou may alias $(D map)   with some function(s) to a symbol and use\nit separately:\n\n----\nalias map!(to!string) stringize;            \nassert(equal(stringize([ 1, 2, 3, 4 ]), [ \"1\", \"2\", \"3\", \"4\" ]));\n----\n",
"line" : 367,




I'd like to get the sections broken out and as many macros expanded
as possible, so the summary text in search results can look pretty.


One option is doing my own parsing, but Walter said the recursive
expansion in the macros was actually very tricky to get right.. so
I'd prefer to skip doing that again myself if I can.


> This generator does not give much information btw, but
> my impression is that people do not use it that much and,
> consequently, no feature request is emitted.

Yeah. It isn't bad though; more detailed than dpldocs.info
version 1.0 which did nothing more than grep '<a name' in
the generated html!

Am 01.08.2012 16:07, schrieb Adam D. Ruppe:
> I'm (finally) updating my dpldocs.info website again, and
> before I just pulled names from the generated Phobos html.
>
> This time, I want to use dmd's json output to avoid dependency
> on the specific html layout of std.ddoc.
>
>
> Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
> one big .json file with all the info.
>
> But, the comments aren't parsed at all.
>
>
>
> So, my question is: do we have a ddoc parser that stands alone?
>
>
> BTW: this is the complete listing from json so far:
> http://dpldocs.info/search/awesome
>
> I'm actually impressed with the speed, considering it is re-parsing
> 3 MB of generated json per load. std.variant and std.json might not
> be speed demons, but they are plenty good enough for me.
>
> The ultimate goal here is to revitalize my D search engine and to
> make it as a nice index for all functions, with brief descriptions,
> etc., hopefully for all D libs everywhere.
>
> With the html, dealing with different people's macros to parse it
> is a pain. But with the json, dmd always does it the same way, so
> with some luck, I can just add a box "upload your project's json
> file" and allow easy instant submissions for other libs.

I have a simple DDOC parser in vibe.d (*). No guarantees that the output is always acting the same as DMD (e.g. there is no support for special sections yet), but it was good enough for me so far.

Something like this should work:

import vibe.vibe;

void main(string[] args)
{
	auto fin = openFile(args[1], FileMode.Read);
	auto ddoc = cast(string)fin.readAll();
	auto html = appender!string();
	filterDdocComment(html, ddoc);

	auto fout = openFile(args[2], FileMode.CreateTrunc);
	fout.write(formatDdocComment(html.data));
}

It doesn't really have dependencies to vibe.d so ripping it out should also be easy.

(*) http://vibed.org/api/vibe.textfilter.ddoc#filterDdocComment

...that module is not really documented atm

On Wednesday, 1 August 2012 at 15:09:45 UTC, Sönke Ludwig wrote:
> It doesn't really have dependencies to vibe.d so ripping it out should also be easy.

Thanks! It looks like that will do the job.

On 8/1/2012 8:20 AM, Adam D. Ruppe wrote:
> On Wednesday, 1 August 2012 at 15:09:45 UTC, Sönke Ludwig wrote:
>> It doesn't really have dependencies to vibe.d so ripping it out should also be easy.
> 
> Thanks! It looks like that will do the job.

I'd almost pay money for the ddoc support that's inside dmd to be pulled into a separate process that just uses the json output.  I suspect it wouldn't be that hard to separate them, just never have had the time to try it.

On Wed, Aug 1, 2012 at 4:46 PM, Adam D. Ruppe <destructionator@gmail.com> wrote:
> On Wednesday, 1 August 2012 at 14:36:26 UTC, Philippe Sigaud wrote:
>>
>> Well, the comments are discarded as soon as lexical analysis is done, I guess.
>
>
> They are actually there if you do both -X and -D together. (-X alone doesn't show it; it must be discarded then, but -D keeps it).

Ah, dammit. I was at work far from a D compiler. I never used -X *and* -D at the same time.

>
> The problem is though that it is the text from the comment without any organization or macro expansion.

OK.

>> This generator does not give much information btw, but
>> my impression is that people do not use it that much and,
>> consequently, no feature request is emitted.
>
>
> Yeah. It isn't bad though; more detailed than dpldocs.info version 1.0 which did nothing more than grep '<a name' in the generated html!

No no no, I wasn't talking about your app! dpldocs is cool!
I was talking about the json generator in DMD, which, to my eyes, does
not give enough information yet to be truly useful.

On Wed, Aug 1, 2012 at 8:07 AM, Adam D. Ruppe <destructionator@gmail.com>wrote:

> I'm (finally) updating my dpldocs.info website again, and before I just pulled names from the generated Phobos html.
>
> This time, I want to use dmd's json output to avoid dependency on the specific html layout of std.ddoc.
>
>
> Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got one big .json file with all the info.
>
> But, the comments aren't parsed at all.
>
>
>
> So, my question is: do we have a ddoc parser that stands alone?
>
>
> BTW: this is the complete listing from json so far: http://dpldocs.info/search/**awesome <http://dpldocs.info/search/awesome>
>
> I'm actually impressed with the speed, considering it is re-parsing 3 MB of generated json per load. std.variant and std.json might not be speed demons, but they are plenty good enough for me.
>
> The ultimate goal here is to revitalize my D search engine and to make it as a nice index for all functions, with brief descriptions, etc., hopefully for all D libs everywhere.
>
> With the html, dealing with different people's macros to parse it is a pain. But with the json, dmd always does it the same way, so with some luck, I can just add a box "upload your project's json file" and allow easy instant submissions for other libs.
>

There is already a pull request to add (at least some) ddoc to json:

http://forum.dlang.org/post/ycfyhcsbciguzaxwuppv@forum.dlang.org

On Wednesday, 1 August 2012 at 17:52:02 UTC, Brad Anderson wrote:
> There is already a pull request to add (at least some) ddoc to json:
>
> http://forum.dlang.org/post/ycfyhcsbciguzaxwuppv@forum.dlang.org

Well, we already have some... but that request looks like it
does a lot of good stuff.

On Wednesday, 1 August 2012 at 17:40:13 UTC, Philippe Sigaud wrote:
> No no no, I wasn't talking about your app! dpldocs is cool!

Thanks! I think the new version will be even better.
Preview:

http://dpldocs.info/search/search-2?searchTerm=post
http://dpldocs.info/search/search-2?searchTerm=get

I'm now extending the search to be able to look at
the type names from the json and do more inter-linking.
It also shows the brief description, if present, and
will filter out private members for you.

And, of course, it will be able to look at submitted
libraries too.

The little breadcrumbs I intend to make into nice indexes,
giving some nice sorting that most ddoc sites don't currently
do.


Still a ways to go, but I think it will end up really nice.

> I was talking about the json generator in DMD, which, to my eyes, does not give enough information yet to be truly useful.

Yea, but it's still better than nothing at least.