Markdown, Asciidoc, or reStructuredText - a tale of docs-as-code

Who writes the developer documentation for your product? During the early days of my tech career, I worked on a database replication product at IBM. My immediate team had developers and testers but not a single technical writer. I can speak for myself when I say that I never bothered to know who keeps writing the lengthy and complex documentation for every release. Since then, many companies in tech (including companies like IBM) have taken a modern approach to documentation. This "modernization" started when more and more developers started writing documentation (not all developers hate writing docs 🤭). Having git running in their DNA, these developers wanted to bring in collaboration, version control, scripting, and easier bug tracking within developer documentation.

ℹ️ Note that I'm not using the term "product documentation", rather "developer documentation". If your users are developers, it's more likely that they would appreciate reading docs written by other developers.

With the text files for documentation in version control system and changes going through the pull request process, the documentation is treated exactly like software code and hence docs-as-code. I see five stages in docs-as-code implementation:

1. Write: Use a markup language to write the docs.
2. Extend (optional but almost a requirement): Use a framework to avail features that are not part of the basic markup.
3. Host: Host the markup files on some repository. This could be self-hosted or a managed provider.
4. Automate the build (optional but highly recommended): Add a CI with spelling, prose, link checks, etc.
5. Publish: Deploy your documentation live using platforms like GitHub (Pages), Vercel, or Netlify.

In the first part of my docs-as-code series, I'll talk about the choice of markup languages, the available frameworks, and do a comparison among Markdown (md), Asciidoc (adoc), and reStructuredText (reST) based on some use cases. I'll skip the history but in case you're interested, my friend and colleague Tibs delivered a talk on the history of text markup languages. My hope is to provide you with a detailed analysis of these choices of markup languages and frameworks so that you can make an informed decision when selecting one for your next developer documentation project. If you don't need a brief intro and would like to skip the appetizer, you can go straight to the entrée - Choose the right markup section.

The choices - md, adoc, reST

Before choosing a markup language, you'll need to choose either a desktop code editor like VSCode or stay in your terminal and use something like EmacsVIM. Most code editors will have plug-ins for popular markup languages which will offer you syntax highlighting, preview, etc. If you're testing or learning a specific syntax, there are many online editors, such as, StackEdit for Markdown.

The main choices for document markup languages are:

• Markdown (md)
• Asciidoc (adoc)
• reStructuredText (reST)

Markdown (md)

The original version of Markdown by John Gruber doesn't specify the syntax unambiguously. For example, it doesn't have table formatting. Due to the popularity of Markdown and the limitations of Gruber Markdown, a plethora of variations, or "flavors", of Markdown have been introduced. Due to this lack of standard, folks like Eric Holscher, who are passionate about the community, have been vocal about not using markdown for documentation.

In 2014, a group of Markdown fans came together and established CommonMark - a standard, interoperable and testable version of Markdown. In March 2017, GitHub published a formal spec for GitHub-Flavored Markdown (GFM); based on CommonMark. In the accompanying blog post when releasing GFM, GitHub addressed many of the limitations that Eric Holscher raised, things like how many spaces are needed to indent a line, or how many empty lines you need to break between different elements. GFM is, by far, the most popular flavor of Markdown.

If you're just starting out with Markdown, check out the syntax and write some hello-world docs online.

For Markdown, there are A LOT of flavors. Despite all eleventy-zillion flavors, Markdown is loved for developer documentation due to its simplicity.

Asciidoc (adoc)

You started with Markdown, but then you needed to embed a YouTube video. So you embed HTML within your Markdown doc. As your project grows, there will be more HTML and some JavaScript embedded in your Markdown files. With out-of-the-box features like tables, admonitions (tips, notes, warnings, etc.), and table of contents, Asciidoc is a popular choice for documentation projects that want to avoid gluing HTML and JavaScript for advanced use. Asciidoc has no flavors so there's really just a single standard. You might have heard about GitHub Flavored Asciidoc (GFA). This is not a flavor per se, rather GitHub using Asciidoctor in safe mode to render files with the extension .adoc, .ad, and .asciidoc. More on Asciidoctor under the framework section. asciidocLIVE is a great place to start if you're testing waters with Asciidoc and would like to learn the syntax.

reStructuredText (reST)

If you're in the Python community, you might be using reStructuredText for documentation. A popular site that uses this markup language is WriteTheDocs. The reStructuredText parser is a component of Docutils and is the default markup language used by Sphinx. reStructuredText can be extended using custom directives that can satisfy a wide variety of documentation needs. Here is a quick primer on reStructuredText if you're getting started. You can play with the reStructuredText syntax using this online editor.

reStructuredText offers a number of useful directives out-of-the-box. For example, admonitions ("safety messages" or "hazard statements") can appear in reST like this:

.. DANGER::
   Do not review/merge your own PR!

You can also use sidebar, math, and a number of other useful directives.

Framework Framework Framework

For a serious documentation project, you're almost certainly using a framework for specific features and to extend the limit of the vanilla markup. In this section, let's check out some of the popular frameworks for Markdown, Asciidoc, and reStructuredText.

Markdown frameworks

Considering that the end result of a documentation project is often a static site, the words static site generator (SSG) tool and Markdown framework can be used interchangeably. Here are my top five picks:

1. Jekyll, the engine behind GitHub Pages, was the most popular SSG until Hugo came out. Written in Ruby, Jekyll is a great choice for writing blog sites. Jekyll can take your content written in Markdown and Liquid templates and render them to a static website deployed to a server of your choice. The site you're currently on is built using Jekyll and GitHub Pages.

2. Hugo is probably the most popular open-source static site generator. Written in Go, Hugo builds sites at a blistering speed. It has a native support for a variety of content types, menus, and dynamic API-driven content. These don't require you to use an additional plugin. You can select a theme that suits your project from a wide selection. Check out some of the project websites that are built with Hugo.

3. Released in 2022, Markdoc is a relatively new Markdown-based authoring framework. The Markdoc project is open-source and it powers Stripe's documentation. Their website has a live edit button which makes the website a playground for you to give Markdoc a try. Documentations created with Markdoc will automatically render with your React app and using @markdoc/next.js for your Next.js app.

4. Configured with a single YAML file, MKDocs is the fourth Markdown framework on my list, which falls in the category of SSG. Although there are not as many as in Hugo, MKDocs offers a few official themes and a number of third party themes. As a Python-based framework, you can use pip to install MKDocs plugins. You can follow this getting started guide for your first MKDocs project.

5. Last, but certainly not least, among my favorite frameworks is the family of frameworks based on MDX. Before that, let’s understand what is MDX and how does it vary from MD.

According to mdxjs.com:

JSX is an extension to JavaScript that looks like HTML but makes it convenient to use components (reusable things). MDX allows you to use JSX in your markdown content. You can import components, such as interactive charts or alerts, and embed them within your content.

With over 40K GitHub starts, Docusaurus is undeniably one of the most popular modern documentation frameworks based on MDX. Docusaurus orignates from Meta Open Source (f.k.a Facebook) and is packed with content-centric features (docs, blog, pages, versioning, i18n, a11y, SEO...). As a React + Node.js static site generator, you can either use MD to get started easily or use MDX to make your docs look interactive. This showcase of documentation sites using Docusaurus is enough to convince me that the UX and DX of a documentation site is as important as the content itself.

From the creators of NextJS, the next SSG based on MDX is Nextra. Nextra offers advanced syntax highlighting, ease of i18n creation, out-of-the-box full text search, and Markdown link and image converted to Next.js Link and Next.js Image. With Nextra, you can write a blog or docs using the themes available. Here are the sites that are built using Nextra.

Besides these five frameworks, Notaku gets an honorable mention that uses Notion as CMS to create documentation site. With SEO and site performance optimization out-of-the-box, Notaku can be a great choice for teams those are already using Notion.

Asciidoc frameworks

1. An implementation of the docs-as-code approach, docToolchain is a collection of scripts that makes it easy to create and maintain powerful technical documentation. It is a popular open-source project that uses jBake under the hood as the SSG. docToolchain can publish to Confluence, generate PDF using an Asciidoctor plugin, and more.

2. Asciidoctor is a Ruby-based text processor for parsing AsciiDoc into a document model and converting it to HTML5, PDF, EPUB3, and other formats. Built-in converters for HTML5, DocBook5, and man pages are available in Asciidoctor. Asciidoctor has an out-of-the-box default stylesheet and built-in integrations for MathJax (display beautiful math in your browser), highlight.js, Rouge, and Pygments (syntax highlighting), as well as Font Awesome (for icons). Although Asciidoctor is written in Ruby, that does not mean you need to know Ruby to use it. Asciidoctor can be executed on a JVM using AsciidoctorJ or in any JavaScript environment (including the browser) using Asciidoctor.js. You can choose any one of three Asciidoctor processors (Ruby, JavaScript, Java/JVM) and get the same experience. You can also use the Asciidoctor Maven Plugin to convert your Asciidoc documentation using Asciidoctor from an Apache Maven build.

3. Unlike docToolchain or Asciidoctor, Antora is a true framework for Asciidoc that can store, retrieve, and aggregate all Asciidoc content from multiple git repositories. Antora’s page referencing system isn’t coupled to filesystem paths or URLs. You are able to cross reference pages across a local machine, a staging environment, and a production environment. To generate a site with Antora, you need the Antora CLI and Antora site generator.

reStructuredText Framework

If you've noticed the usage of "framework" instead of "frameworks" in the above heading, you can guess that there's only one framework for reStructuredText. Sphinx, originally created for Python documentation, takes plain-text files in reStructuredText format and transforms it into HTML, PDF, and any output formats. A few well-known projects that use Sphinx for documentation are Django and ReadTheDocs.

Sphinx is incredibly powerful and can offer a table of contents, automatic links for functions, automatic code highlighting using Pygments, and other capabilities using built-in or third-party extensions. If you'd like to use (a flavor of) Markdown with Sphinx, you can do so using MyST-parser - a Sphinx and Docutils extension to parse MyST.

However, many folks complained about the difficulty in surfacing and debugging errors that happen in the Sphinx build process. The fact that Sphinx build is significantly slower than other SSG frameworks, the development and writing flow takes a hit in the process.

Choose the right markup and framework

If you're a developer, learning the syntax and styling of a markup language is relatively easy compared to learning a programming language. When you learn the syntax of a markup language and can render an HTML page locally, you reach the hello world nirvana of documentation. However, the difficulty doesn't necessarily start when you write the first documentation file. As you try more advanced tasks for your developer documentation project, you lean more on the framework community behind the framework.

I asked on Twitter about the choice among Markdown, AsciiDoc, and reStructuredText and the first version of this blog summarized the findings. When I published this blog back in November 2022, it got quite a few attentions and was trending on the first page of HackerNews. Besides the satisfaction from the blog's success, I felt terrible for missing out on tools like Jekyll, docToolchain, and Docusaurus. Thanks to Dan Moore, Ralf D. Müller, Sebastien Lorber, and the HackerNews commenters, I've updated this blog to include the missing tools.

Scripting language preference

The first line says it all! Almost every single question of "Why reST?" is answered with "because we have a bunch of Python devs...".

Next is Markdown. If your project uses some form of workflow and you'd like to run some scripts within your Markdown, your choice of scripting language might vary from JavaScript (Jekyll, Markdoc, or Docusaurus) to GoLang (Hugo) to Python (MKDocs).

Unlike reStructuredText or Markdown frameworks, Asciidoc's Antora framework provides extensive feature set out-of-the-box so that you don't need to add scripting languages like Python, Ruby, or JavaScript within your documentation. Full disclaimer that my take is based on Antora's website. An advanced user of the framework might be able to confirm this claim.

Syntax

Let's look at a few syntax comparisons to understand how simplicity helps Markdown win many projects.

Setting aside the annoyance of having to hold down the “=”, "~", or another underline-like character a bunch of times, the big problem with reST headers is that header characters in reST have no single mapping onto the header hierarchy.

Similar to reST, critics say that AsciiDoc is less familiar to engineers and technical writers. AsciiDoc is said to have a number of syntactic idiosyncrasies, such as the use of multiple leading asterisks in order to express nested bulleted lists or varying the length of the delimiter line to nest delimited content blocks.

Extendability

Many tech companies love the rich ecosystem of Remark to transform Markdown with plugins. Still, extendability is the subject where Markdown historically scored the lowest. Most folks mention that Markdown is not suitable for serious documentation projects. Some teams might be tempted to include custom HTML and CSS within Markdown and then get stuck in a messy situation during migration to another framework.

Let's think for a moment here 🤔 ...

How often do you migrate your docs? Migrating any mature documentation project that uses some sort of framework is never easy. If you choose the wrong framework or overload your project with custom HTML/CSS/JS without taking advantage of a framework's built-in feature, Markdown is not to blame here.

If you're migrating your documentation project from one markup to another, you'll want to know about projects like Pandoc. If your build process includes a process to convert markup, Pandoc can be run with GitHub Actions. If you're planning on converting Asciidoc files to Markdown, you'll have to take a detour via the XML route. The proces will be: Asciidoc ---> XML ---> Markdown.

MyST-parser, which is both completely compatible with CommonMark and also supports all ReStructured Text directives, can be used to convert reST to Markdown.

Internationalization (i18n)

Internationalization (i18n) pain for a documentation project is a process problem, not a feature gap. Documentation frameworks are not meant to translate your developer docs for you into the language of your choice. Some frameworks might offer i18n support, like the Crowdin support in Docusauraus v2. With Jekyll, you have to pick a theme like this one. I doubt if the reST or adoc frameworks would differ much from this process. Besides the framework support, you'll need a localization vendor like lokalise or a community driven platform like Crowdin to actually do the translation. Sébastien goes into great detail in this Docusaurus RFC when discussing i18n support in Docusaurus v2.

Other factors when considering a markup

If you need to maintain different versions of your documentation based on product releases, you need to carefully plan in advance. Copying file snapshots after every release is a BAD IDEA. This does not scale, and pretty soon you'll exhaust your resources. The ideal way is to use docs-as-code and use git to manage the versions of your documentation.

Do you have parts of the documentation behind a firewall that require authentication to access them? You'll need to set up an identity and access management (IAM) service in front of your static website. Unless IAM is your jam, it's better to avail of a managed service to tackle it.

Authoring Experience ⚖️ User Experience

Let's look at some documentation sites with great UX:

Humble brag that the last one in the list (Aiven 🦀) won 2022 DevPortal Awards in the category Best DevPortal Beyond REST Platforms. If I added a list of great docs websites but didn't include the OG, it would be a crime. Check out one of THE BEST documentation out there - Stripe Docs, built using Markdoc.

What makes a great user experience for a documentation site? The factors affecting your users are:

1. general look and feel of your website
2. proper use of interactive components (although this is completely optional)
3. ease of finding information (including the correctness of the information)
4. features like copy code to clipboard, syntax highlighting on code, etc.
5. how less annoying your site is with things like pop-ups, tracking, etc.

I like that you care deeply about your users. How about the folks who work really hard to build these docs? If the emoji is loading properly, you'll see that I put a scale icon between authoring experience and user experience on the section heading. Here are the factors affecting your authors:

1. ease of creating their first PR (you are using docs-as-code approach, right? RIGHT?)
2. maturity of the content architecture (Can I easily guess which sub-folder my new file will go under?)
3. level of scripting knowledge required to contribute (do I need to be a react dev to contribute to the docs?!)
4. automation within the content review/publication process to catch linting, spelling and other errors (this is not related to the markup/framework)
5. frequency of large migration projects (ideally "zero")

Points 2, 4 and 5 don't relate to the choice of the markup language or framework. Depending on points 1 and 3, your docs contribution will include almost anyone from your company or only the DevRel team and technical writers.

Writing documentation is different from writing code.

Writing documentation is different from writing code.

Yes, I needed to repeat that line. As a user, I love the visually appealing documentation sites built using an MDX-based framework. While MDX affords users more power and flexibility, it comes at the cost of complexity. Content can quickly become as complex as regular code, which can lead to maintainability complications or a more difficult authoring environment. You can choose an MDX-based framework for other features and stay away from overscripting. But the temptation (to add more scripts) is harder to resist than one might think.

With the above discussion, can we find a middle ground that balances both the user experience and authoring experience? While all 10 factors affecting user and authoring experiences are critical, some of the ones I mentioned above don't relate to your choice of markup and framework (for example, content architecture or if you add too many pop-ups). I'll highlight five questions you can ask when selecting a markup and framework.

Question 1: Do you have a strong reason for NOT using Markdown?

Question 2: How much do you care about the appearance of your docs site? Do you need a beautiful and interactive one that you can maintain over time?

Question 3: What are the features that matter to your users and authors? Search? Internationalization? Versioning? Pick the framework that delivers (most of) these features.

Question 4: Do you have a special need, like building docs from multiple git repositories or requiring authentication on certain parts of the documentation?

Question 5: Is the framework open-source?

Wrap up

In this blog, I covered three popular markup languages, some very popular frameworks, and my opinion on the choice of markup language and framework based on a few key factors. If there's one thing you want to take away from this blog, it will be to find the balance between the ease of getting started (authoring experience) and the ease of finding information (user experience). If you liked this blog or have any feedback, please reach out.

Appendix

Since this blog had its day of fame on HackerNews, it has generated a lot of insightful discussions. It would be a shame not to capture at least some of them.

Addison wrote:

I have spent a lot of time looking in this space recently for helping to revamp documentation and I really really have fallen in love with Markdoc.
Markdoc just hits the sweet spot of being super easy to get started with but elegantly extensible that makes it scale. I think the OP here simplifies a bit though of what Markdoc is. While it is pretty simple to integrate into a next.js site for a SSG doc site, it is more of a library that can be integrated into almost any site or rendering framework.

In some ways, this is the biggest "challenge" of Markdoc right now. It isn't focused on a polished out-of-the-box experience like Docusaurus or MKDocs, but is instead more of a DIY tool.

That said though, what is there is really great. With the ability to create custom tags easily and then the ability to analyze and transform an AST in a simple, but easy to understand way, I think markdoc is actually a great option for more than just building a doc site, but as a more general purpose tool for authoring any text-heavy content.

With Markdoc, I have built:

• a higher level utility for creating a "library" of content with consistent ids for stable and validated links
• a validation library to ensure that doc structures follows best practices like having metadata tags in the frontmatter, properly nests headers and doesn't skip H3s, etc
• an integration for authoring and reusing doc content in spectacle presentations
• a clear direction of how to "scale" docs-as-code as we were struggling to do that with a simple, flat file of markdown files

I have started to toy with the idea of a more general purpose CMS built around markdoc... but in general, a really great tool and kudos to stripe team for building it :)

Anonymous1 wrote:

I've done a lot of research and testing with Markdown, Asciidoc, and reStructuredText to see which would work best for my company's documentation needs. We ended up going with Asciidoc and Antora for the following reasons.

Asciidoc:

• Almost as simple as Markdown.

• Less convoluted than reStructuredText.

• Excellent support for complex tables, captions, callouts, etc.

• We prefer Asciidocs table structure to Markdown's since it is easier to create and maintain.

• Excellent documentation.

Antora:

• Comes with a default template, which makes building prototypes easier.

• Ability to pull from multiple git repositories.

• Native Asciidoc support.

• Fast compile times.

• Good documentation.

Based on our research, I even migrated my personal 11ty sites from Markdown to Asciidoc and have been quite happy with it.

Anonymous2 wrote:

I tend to agree with the author's sentiment re: Asciidoc, but that is a subjective impression, driven by my industry and the requirements it imposes. (note that the first version of the blog suggested Asciidoc/Antora as the ideal choice).

Here. Let me tell you a story.

Some years ago, Leadership in decided to stop paying for the S1000D[1] software. This would make the publications group effectively homeless. The objective of this tactic was to force a move to the PDM system's tech writing system, which had previously been rejected by more or less everyone.

I put out a plan to keep the S1000D architecture (filenames, books, links, etc) but use lightweight markup and open standards from the programming industry, to do the actual writing. Which lightweight markup language? I needed the following:

• Transclusion, need to bring in files (data modules) from a centralized publication module

• Partial Transclusion, need to be able to bring in part of another file, what would be called CIRs in S1000D

• Conditional content, usable inline, so that a step or a figure could be toggled on and off depending on a condition set in the Publication Module. This emulates Applicability in S1000D.

• Complex print output

• Nice-to-haves: an AST that maps to a legacy format aka DocBook/DITA/S1000D/MIL-STD-38784/etc; publishing pipelines capable of weird crap like complex front matter, TOC, indices, header/footer/margin running content; a singular standard that had some life in it

I took pretty much every lightweight markup variant out for a spin with a test migration and publish, on my own time, nights and weekends. I found that there was a Markdown variant that could almost do everything I wanted, but it had a dependency on a document processor. ReST functionality similarly depended on notebooks and Sphinx. LaTeX HTML pipelines were a bit janky and hard to set up on Windows. Neither really tied to a legacy XML format, which limited round-tripping with something as insane as S1000D. And the print options for both ReST and MD were not where I needed them, particularly for tables, but also for running content. Asciidoc came the closest to checking off all the boxes for me.

Anyway, it happened, it worked, I did it. But it hit business process problems. Stuff that would have hit any new instance of a pubs too. Like "we don't actually know which plane parts can work with each other" or "SMEs will never ever ever do a review in a text editor . . or electronically . . or in anything but a dead tree" or "a few writers don't know what the scroll wheel on the mouse does" or . . eh, take your pick. I ate my own gun, though, and left. They're still using it - and probably still cursing my name - to this day. But that had been one of my big goals: S1000D architecture on lightweight markup.

So. Lightweight markup. I haven't continued to shop around, and I probably should. For one, I'm of the growing opinion that inline conditional content is a mistake as a general design, and that the conditionals should probably live in the processing layer. That's a big chunk of the requirement covered by Asciidoc's include directive. For another, I won't deny that the Markdown ecosystem is about 100x the Asciidoc ecosystem, due to the much greater dev count with the JS ecosystem vs the Ruby ecosystem. Some of the toys in Markdownland are worth the move all by themselves. And round tripping S1000D? You can't even round trip S1000D with itself.

Then again, there's so many damn MD variants. .

[1] XML vocabulary for documentation of mil/aero systems. Think DITA but a million times more complicated. And a new Issue every two years, without any commonality in the scope.