Lets take a break from setting up CouchDB in Rookeries, and discuss
documentation.

I recently made the switch to using Markdown for the majority of the prose
style documentation for Rookeries. Originally I wanted to support both
reStructuredText and Markdown. However for reasons I’ll write about, I will
concentrate on supporting Markdown in Rookeries.

Requirements

What do I expect from documentation for Rookeries? I want an automated setup
that will allow for easily writing prose documenation, API level documention
and also that the documentation can act as a test fixture as well. There is no
need to duplicate efforts maintaining two sets of documentation: one for the
code and one as a test sample.

Avoiding Duplication – Unifying Documentation and Test Fixtures

Some of the tests for Rookeries, require actual content living inside a
database. This is an excellent way to dogfood Rookeries, by forcing it to
handle some of the content it will have to support. Currently the test
fixtures live separately from the documentation. However the actual fixture
refers to the path for the sample files, as part of its setup. Whether this
path points to the test fixture folder or any other other folder in the
Rookeries source tree is arbitrary. So why not have the same documentation as
both project documentation and sample test data?

Keeping API Documentation

However I still want API documentation, not only for my sake but to allow
future contributers to extend Rookeries or build plugins for it. Going with
simply just the prose documentation is not enough. So I wanted to keep my
current Sphinx autodoc setup, or have something similar parse docstrings in
my code and generate gorgeous API documentation.

Result

After some trials, I hit upon a way to support both Markdown and
reStructuredText in my Sphinx powered documentation. The Markdown files are
being referenced in the tests, so all my requirements were met. Huge thanks
to Eric Holscher (dev on the ReadtheDocs sites) for figuring this out
originally.

Alternatives

The other alternatives were either use a Markdown-only documentation
generator (mkdocs) or support reStructuredText in Rookeries.

What not Use mkdocs?

mkdocs is an awesome project that uses Markdown to generate documentation via
Sphinx. mkdocs works remarkably well, has a few nice themes, abstracts away lots of the Sphinx configuration and has a nice workflow for writing the documentation. However its API documentation story is lacking.

State of Autodoc API Documentation

While the initial impression I got from mkdocs was excellent. However when I
tried to use mkdocs for generating API documentation I ran into problems. Judging by the roadmap of mkdocs, there is not much desire to support API documentation. There is an experimental project to hook up mkdocs to the Sphinx autodoc.
However this did not work for me.

Why not Support reStructuredText in Rookeries

Alternatively, I considered going in the other extreme of only supporting
reStructuredText. Aside from the fact that RST syntax is not always the
easiest to remember, I ran into some more technical challenges.

State of RST Frontend Clients

The first major issue is that there are no reStructuredText frontend clients.
There are lots of Markdown clients for Javascript, but none for RST. While I
do plan on rendering content on the server side, I do would prefer to have the
option of doing some of the rendering on the client side.

Working with Docutils in Python

Less of an issue, but more of encumberance is working with reStructuredText in
Python. The docutils library is the standard way to convert RST into a number of formats. The documentation for docutils is horrible. I wish could be more charitable but it took me a good 30-45 mins poking around the docs to figure
out how to programmatically render reStructuredText into HTML:

import io
from docutils import core as docutils_core

with io.open('my_rst_sample.rst') as src:
    beta = src.read()

doc_core.publish_string(beta, writer_name='html')

Mind you this only gets you as far as having a full HTML documentation, that
you need to muck around with. I did get partial HTML rendering earlier on
Rookeries’ history, but it was not obvious or simple to get to. I don’t need
docutils’ grand, book-publishing ready setup nor their command-line tools. I
just need something to render marked-up text into HTML for a blog.

In the future I might wade into dealing with reStructuredText, but I will pass
on that for now. There are more important open issues with Rookeries than
which exact format to use.

Use pandoc

When the topic of rendering between markup format comes up, so does using
pandoc. Pandoc is a great tool for converting between various markup and document formats. And it does a decent job of translating RST to Markdown and back again:

pandoc -f rst -t markdown sample.rst -o sample.md

Now I don’t plan on relying on Pandoc for Rookeries. But I might require it
when I need to import and export data from non-Rookeries blogs and sources.

Markdown in Sphinx

The solution I finally settled on was making Sphinx render Markdown. I found a great article on configuring Sphinx to handle both reStructuredText and Markdown.

Setting up Sphinx to Render Markdown and reStructuredText

The setup is fairly straightforward. Install the remarkdown library that adds
Markdown support to docutils which Sphinx will use:

pip install remarkdown

Next add the following configuration to the Sphinx conf.py configuration:

from recommonmark.parser import CommonMarkParser

# The suffix of source filenames.

source_suffix = ['.rst', '.md']

parsers = {
    '.md': CommonMarkParser,
}

Viola! You now can mix and match Markdown and reStructuredText in your
Sphinx documentation. I would stick to RST when dealing with more
complicated macros for Sphinx (like the releases changelog add-on I use).

Markdown in WordPress

A final note: I want to work toward transitioning this site to Rookeries. So
I started playing around with how it would be to write all my blog posts in
Markdown. I am using Sublime Text for my editor. On the WordPress side, I
found that WP-Markdown is a nice WordPress plugin for writing content in
Markdown and then rendering it to HTML.

…and we’re back!  Or rather the site is, thanks to Eric who helps admin the VPS that this site runs on.

So much has happened in the meantime: PyCon Montreal, furthering my experience in working on Python microservices + Docker + Ansible, my dabbling in the startup and JS worlds.  And life in general, with friends getting married and life in general moving forward.

One of the lessons learned in this outage, is to keep better backups and use automated configuration managers when administrating a site.  I’d love to talk about my Ansible playbooks that are just now approaching the point where I have almost completely automated backups and deployments.  But I’ll do so at another time.

Just a quick post this time, since this is one heck of a busy day.  Actually the whole year has been kept me super-busy beyond my wildest dream.  While I could of made do with less stress, it definitely pushed me to grow out of necessity.  I definitely am a stronger professional than in the beginning of the year.  I can consider myself a senior developer and be confident in my technical, communication and inter-personal skills.

And I learned to dance, no kidding.  I learned to do things I did not imagine I could do.  I feel that I am calmer, stronger and better person overall.  Of course I can not sit on my laurels.  There is still so much to do, learn and explore.  I look forward to this coming year.  Hopefully you do too.

I wish everyone a Happy Belated Christmas and Happy New Year!  See you all in 2013!