Overview

marmos (MARs + deiMOS... I'm not creative) is a tool for generating a "generic documentation model" from D code that can then be consumed and converted into other formats for external documentation generators.

It's in an alpha state right now since it's in a half-rushed state where it captures a fair amount of the basic structure of D code, but still has a lot to be desired, but I got excited enough about the current result to make an announcement.

In more specific terms:

• It uses dmd-as-a-library to parse individual D files (only syntax analysis).
• It generates a "strongly-typed" JSON file. Each object has an "@type" field denoting its layout.
• It maintains a separate documentation-focused AST so that future changes to the frontend's AST can be mitigated. At some point I'll add versioning to the documentation types, but for now I'll wait until a stable release before thinking about that.
• A core part is integration with external tooling, so a goal for marmos is to generate typings and parsers of the documentation model for other languages, to make it easy for them to consume the JSON output.
• (Currently only a rushed TypeScript generator exists, but it does the job surprisingly well).
• It contains a dogfood/reference TypeScript project that generates API pages for Microsoft's docfx tool.
• As it was mainly purpose-written for one of my other unannounced projects, the comment format it supports is some dodgy Markdown-esque format rather than ddoc (I don't like writing or reading ddoc). This is very much subject to change though, it just needs more man-hours put in.

I'll reiterate that this is an alpha, early version release, please keep expectations low :D

Getting Started

It's not easy to get it setup locally yet (something I want to address fairly soon), but the README goes over it a bit, and the demo repo linked below shows a fairly simple setup.

Examples

This demo repo: https://github.com/Juptune/marmos-docfx-demo

Generates the following site: https://juptune.github.io/marmos-docfx-demo/

I'd say this is a decent example of the current generation's (lack of) ability: https://juptune.github.io/marmos-docfx-demo/juptune/juptune/http/v1/Structs/Http1WriterBase.html

The above being generated from this comment, to get a feel for the formatting: https://github.com/Juptune/juptune/blob/master/src/juptune/http/v1.d#L1357

Next Steps

I'd like to get some feedback on the general idea of this tool; whether it's useful at all, etc.

Beyond that I really need to get the missing data collection sorted. e.g. arrays; functions/delegates, and some other things show up as <bug: unknown> since these types aren't properly extracted from the frontend yet... enum stuff as a whole needs properly sorting out when rendering types.

Any pointers on pitfalls the frontend's AST has would be nice as well; e.g. it seems I need to manually handle attribute blocks (private:, etc.) instead of it being auto-propagated onto the child nodes of the blocks?

Also here's a gist of the generated TypeScript stuff, since as it's a generated file it's not in the git repo anywhere: https://gist.github.com/BradleyChatha/46cef68f5081077bf76dea6e8817916f

I also realise I completely forgot to even link to the repo... here it is: https://github.com/Juptune/marmos

On Tuesday, 2 July 2024 at 19:18:38 UTC, Bradley Chatha wrote:

Do I understand that this is something that generates a JSON file for consumption by other doc generators?

It's very cool that you are using the parser as a library!

Can you elaborate on why one would want to use this instead of ddoc or something like adrdox or ddox? Do you think this might be in a state some day to replace the D docs (which I've heard complaints about recently)?

-Steve

On Wednesday, 3 July 2024 at 03:19:12 UTC, Steven Schveighoffer wrote:

Yeah basically, the process is basically (in the case of the reference docfx stuff):

mod/one.d -> marmos -> mod.one.json -> marmos-docfx -> docfx YML files... -> docfx
mod/two.d -> marmos -> mod.two.json ^
mod/ten.d -> marmos -> mod.ten.json ^

Each module gets it's own JSON file, and then these JSON files can be fed into some sort of converter tool (e.g. marmos-docfx) to generate more files for some external tooling.

So in this case, marmos-docfx will create API Pages and Table of Contents files that docfx can natively process.

Yeah I was surprised that it was actually kind of painless to use! The main annoyance was having to go through what felt like 50 layers of nesting to find what I want, but that's likely due to a lack of familiarity with dmd's source.

The annoying part was getting Meson setup to compile dmd-fe... I did it before a few years ago but forgot that I needed to set a bunch of versions to disable the backend stuff.

For flexibility I guess, and to use existing site generators.

With raw ddoc for example you don't really get an awful lot out of it:

• No search bar
• No tables of contents/navigation
• Other quirks such as it only supporting specific section names (unless that's changed since I last used it). E.g. if I were to do Workflow: I remember that ddoc doesn't generate a header for it, but just renders it as normal paragraph text.

ddox I've admittedly written off mentally due to past bad experiences, without even really considering it for the modern day:

• Back in my Windows days I struggled to even get ddox to compile or work with dub.
• When I did managed to get it to work, it was failing to deal with my project's codebase (can't remember the exact issues), and so wouldn't generate the full site.
• I'd looked over some existing ddox sites and honestly I just didn't like how it looks by default. I remember trying to use some premade theme that looked better but ran into other issues with that as well...
• Do I remember correctly that it requires you to correctly pass in things like import paths/it does some light semantic pass? I feel like I remember getting some issues around that...

It might be better nowadays, but when I used it, it was just pure frustration. I was also still stuck with using ddoc's syntax (prior to modern ddoc adopting markdown-like features).

It did have better handling of some things though, it had cross-referencing stuff that mostly worked out of the box.

adrdox is awesome; handles comments well; has good features, and looks good, however:

• (kinda minor but) Similar to ddox it doesn't use dmd-fe but instead a (forked?) community parser, so if your code uses newer syntax then you kinda just can't get any docs out of it until the community parser(s) are updated.
  • I had an issue like this with the shorthand int a() => 1; syntax for a while as an example.
• I'm wary about its future, especially it's ability to support future changes to D if D and OpenD begin to diverge a ton.
• I think it's unable to support the use case where you can use the same generator for both an API reference, as well as other forms of documentation?
  • I guess ddoc and ddox can support this via standalone .ddoc files.
  • For example, C#'s doc site can support hand written pages as well as normal API reference pages
  • I will definitely note that this is not needed for a lot of projects, but is a deficiency either way.

Ultimately I decided that documentation sites are very subjective, so I set the goal of leaving the core marmos tool to extract all the important bits and pieces, and then leaving it up to the converter tools to format everything into how they want it to look, and letting the end user decide what site generator they wanted to use.

So I started to look into making use of external tooling that already has a ton of features ready to use so long as I could convert D stuff into the tooling's stuff, and docfx seemed like a good fit for a reference/dogfood target.

I initially wanted to explore built in options for generating a dump of the AST + documentation, but my learn post had no answers, and the AST dump had other issues as I mention in marmos' README.

I don't really know how useful I'll be able to get the generation to be in the end, and it's definitely not as good as adrdox yet, but my goal is to basically be adrdox-level of quality but with the bonus of being able to use external generators & themes (rather than being tied down to default ddoc, ddox, adrdox, or having to spend ages on custom CSS styling + JS integration for basic stuff like searchbars).

In other words I feel there's value in having the ability to extract the structure and documentation of D code for external processing purposes, which I don't believe is something that fully exists yet (at least, isn't purpose built for the task)?

I guess in theory it could one day generate the D doc site, by virtue of having an appropriate converter for the models, but I don't think it's all that close to such a thing yet.

And on one other level is that the Juptune project as a whole that marmos belongs to has an underlying theme of me not really wanting to place trust in other people's work in D now, so having my own tooling gives me the level of personal flexibility I'm looking for.

:sweat: Sorry for the mini essay, but I wanted to write my full thoughts down so I can get corrections where needed by people who know better than myself.

On Tuesday, 2 July 2024 at 19:18:38 UTC, Bradley Chatha wrote:

It would be great to support the ability to show differences between implementations for a different versions(..) branches.

I.e., if the function prototypes are different for Windows and Posix it would be reasonable to display both to highlight differences.

For now, as I remember, documentation built only on DDoc version

On Wednesday, 3 July 2024 at 15:48:34 UTC, Denis Feklushkin wrote:

Yeah that'd definitely be cool to have - I've been having a think on how to best make that work in docfx as well.

On Tuesday, 2 July 2024 at 19:18:38 UTC, Bradley Chatha wrote:

I didn't really want to litter with another announce post so soon, so just posting here that v0.3.0 is out now:

• Most of the information you'd probably like to see in docs is now extracting, so it's a lot better to look at.

• General improvements to rendering the documentation within Docfx.

• The demo site will now automatically update anytime I release a new version.

• The flow for using marmos is slightly easier since the typescript project is now on NPM, and there's also a docker image, though it's all still a bit jank.

There's still plenty of issues left though such as:

• A lot of private symbols are still showing up in the docs.

• I still haven't figured out how to actually extract the right hand side of alias ABC = "how do I get this from the AST?"

• I also haven't figured out how to distinguish between an auto variable and an enum so the docs will just say <auto or enum>.