Rookeries v0.9.0 out – New UI and Live Editing of Sites + Juggling jq with JSON Book Update

It has been quite a while since my last update. I apologize for the long silence. I wanted to focus on getting Rookeries up and running, to the degree that I can host a website. Namely I was hoping to update the Amber Penguin Software website and the Juggling JSON with jq book companion website to use Rookeries. Unfortunately I had to re-architecture the UI and bring the CouchDB management of the site and pages up to spec. However I am happy with the overall results:

Interactive Editing of Menus

Rookeries v0.9.0 introduces support for adding, removing and reordering menu items. It took a bit to find a good simple implementation of a drag and drop able list.

]4 Version 0.9.0 introduces interactive editing of menus.

Interactive Editing of Site Headers and Footers

Rookeries v0.9.0 also extends support of editing pages to site headers and footers. It uses the same inline editor for Markdown and HTML, that the pages currently enjoy.

]5 You can now edit site headers in Rookeries

So what else changed?

  • The migration of the UX over to a sliding sidebar is done.
  • Internally redux was replaced with MobX, which is far more maintainable for me.
  • The addition of saving sites and the current selected page.

There is still a lot to do on Rookeries, such as making the CMS more robust and flexible. Adding and removing pages is still outstanding. So is selection of pages via URLs. Marking whether or not changes have been made, to prompt the user to save. Also to check that the site is not in an invalid state. However things are starting to look up.

A few people have inquired about the state of the Juggling JSON with jq book, and when will a sample chapter and table of contents will be available. I apologize for not replying directly to those emails (I will do so shortly), but I hope to get those out in the next few days, and hopefully before the end of the year. Work on Rookeries has essentially ate up all the time that I would of used for working on the book. I think going forward I will work on both in parallel.

Adding Functional (End-to-End) Test to Rookeries

Testing the client side of Rookeries, has proven to be quite a challenge. Not necessarily because testing well-written React JS components is hard. Rather I found it hard to setup a proper and consistent unit test infrastructure to do so. Rather than going through the pain of writing and maintaining functional tests in Javascript, I decided to take a different path.

BDD + Web Testing – Theory and Practise

I wanted to write my tests in a business-domain-driven (BDD) style. While most developers find awkward to use at first, it is a great way to write out the business functionality and features of an app. It also forces you to think about your app in a non-technical manner, and prove to yourself (and others) that it does what you claim it does.

At work my team has been burned by slow and awkward web tests. Namely we worked with the Robot Framework, which uses its own DSL and is very difficult to work just as we wanted. Debugging tests was also quite unpleasant.

Fortunately one of our newest team mates Kevin Qiu introduced us to lettuce (a BDD framework) and splinter (a Pythonic Selenium interface). And we’ve had a fair bit of success describing scenario using these tests. I won’t lie, Selenium is always temperamental. However the current batch of web tests have been very stable, and has very much convinced me that this setup can work effectively.

The Rookeries Take on BDD-style Web Testing

Rookeries usea a similar setup to what we’ve done at work: namely I use splinter to interface with Selenium. However unlike work, since Rookeries uses pytest instead of nose, I ended up using pytest-bdd to provide the BDD framework for Rookeries. Furthermore, I am using the pytest integration for splinter, to provide some of the browser fixtures needed for the tests.

Feel free to check out the functional tests in Rookeries to see examples of the tests. I also highly recommend watching Dylan Lacey ‘s talk about using Splinter for web testing at PyCon Australia 2013.

A Few Wrinkles I Found So Far

  • Dylan suggests using names for inputs when working with web tests. Unfortunately the react-bootstrap components I rely don’t include support for names, something I plan on submitting a patch/pull request for.
  • One needs to run a localized server as part of the functional tests, which makes for a slightly complicated task setup. This is something I need to simplify.
  • I found using the element.text of a container React component works around the issues of text phrases being broken up over a few components.
  • Using ids is the simplest way to find elements, even thought that isn’t what a user would actually use to navigate the site.
  • ipdb does not work very well when debugging tests. Plain old pdb works wonders though. I am considering switching over to using pdb++

Adding a Code Editor to Rookeries

One of the main selling points for Rookeries, is its ability to let the
writer write content using Markdown, and getting a live preview of the result.
Since the user will be writing in code, it would be a better experience to
use a text editor meant for code. Rather than try to undertake a large task
like this myself, I decided to look around for an editor.

Choosing a Text Editor

My three main concerns for a code editor is for it work properly, integrate
nicely with React and have nice syntax highlighting. After a bit of searching
I found out about the following code editors:

Setting up a Code Mirror-React Component

In the end I chose Code Mirror and specifically the react-codemirror project.
The syntax highlighting does not quite work yet, but I will work on that. The
other integration looked more mature, but ultimately did not want to work for

Setting up the Code Mirror component in Coffee-JSX turned out fairly easy:

React = require 'react'
CodeMirror = require 'react-codemirror'

MyView = React.createClass(
    render: () ->
        options = {lineNumbers: true, mode:'markdown', theme: 'monokai'}
        return (
            <codemirror value={ code } onChange={ @updateCode } options={ options }></codemirror>

CSS Gotcha!

Oddly enough just setting up the CodeMirror component did not work right out
of the box. In fact the result was rendering each line as a separated out

 box.  After a bit of digging, I found that the styling was based on the 
pre-existing CSS style provided by Bootstrap... not by Code Mirror!  

The reason turned out to be simple: I had not included the CSS from Code 
Mirror into the app. Since I did not find a CDN with Code Mirror setup, I 
decided to simply include Code Mirror's CSS into my LESS stylesheets for 
Rookeries, based on the example in react-codemirror:

    @import (inline) "../../../node_modules/codemirror/lib/codemirror.css";

This worked well enough, and I've included some of the CodeMirror themes as 

# Outstanding Issues

Currently the text editor works however the setup is missing an important 
feature: syntax highlighting.  In theory Code Mirror should enable syntax 
highlighting based on the passed in mode.  Setting the mode to 'markdown' does 
not seems to achieve the expected result.

Fixing SPDX Expression Warning in package.json

If you ever run into the following warning when installing your NPM package:

npm WARN package.json rookeries-api-client-wrapper@0.4.9 license should be a valid SPDX license expression

That means you have an improper a name for the specified license in your package.json. So what are valid values for licenses? Well… here is the SPDX list of licenses

Using CouchDB in Rookeries – Part 3 – Configuring a Remote CouchDB Server

In the previous instalment of this series I wrote about installing and
managing on a remote server. Now lets talk about configuring CouchDB so that
it can run as a production server. This will not cover CouchDB’s
configuration extensively, rather I will touch on the parts relevant to

Configuring CouchDB

CouchDB can be configured in two way: either by modifying the INI setting files
located in the /etc/couchdb/ or by visiting the Configuration UI in Futon:
e.g. http://localhost:5984/_utils/config.html I went with the route of editing the local.ini file via Ansible.

Configuring Users Authentication on CouchDB

The default installation of CouchDB does not force you to declare and secure
users. Users and user authentication is totally optional. However since I
did not want open up my production database to the world.

Adding an Admin User

I first added an admin user to the CouchDB configuration. This user being the
admin user for the entire CouchDB server, rather than an individual database.
The change consisted of adding a value for the user and password, under the
admins section:

admin = password

If you’re worried about leaving your password in plain-text in the
configuration, then don’t. After restarting CouchDB (via the Upstart service)
this password gets hashed.

Admin Party No More – Enforcing User Authentication

By default CouchDB runs in what is “admin party” mode, meaning you do not need to log in to make admin or user changes on the server. Naturally for a production server that is not something you want to do. So you have to enable requiring user login:

require_valid_user = true

Getting CouchDB to Talk with the Rest of the World

This part is optional, however if you want CouchDB accessible from more than
the localhost you have configure it to allow connections from multiple sites.
I needed this since I use Codeship as a continuous integration (CI) service for Rookeries, and I wanted to run integration and end-to-end tests using my production database server. (In an ideal world I would have a separate CouchDB server just for testing or have a CI that has a local instance of CouchDB.)

Binding Addresses

The trick to allowing this is to set the right binding address for CouchDB.
This can be done by changing the bind_address value in the httpd section of the configuration as such:

bind_address =

By default this just localhost or You can also setup the
configuration differently. One thing that I am not sure of is passing a
range or a list of different bind addresses. I am not sure this is possible
based on the documentation that I have seen.

What about HTTPS?

CouchDB has options to handle HTTPS and SSL natively. I personally have not configured my site to use HTTPS, since none of my sites
do so currently. Getting certificates and everything setup for all my sites
is a bit involved so I have avoided the issue for the time being. I plan on
getting around to do so in the future.

However if you have the time and option to setup HTTPS, please do so! Putting
up another layer of security around a production CouchDB will help. More
importantly HTTPS gives you and your end users a degree of privacy, that is
rare in these Post-Snowdown times.

Using CouchDB in Rookeries – Part 2 – Setting Up a Remote CouchDB Server


In the second instalment of my series on adding CouchDB support to
Rookeries, I’ll be talking about how I provisioned CouchDB on my remote

Now it sounds counter-intuitive why I would talk about creating and
populating CouchDB databases first before writing about installing
CouchDB. The reason for this backwards step, is that I already have
CouchDB installed locally. At my daytime job at
we use CouchDB extensively, so I already have
CouchDB installed locally on my workstation. I have also worked with the
Operations team to provision CouchDB servers. However it is a different
story when trying to provision and configure CouchDB yourself on your
own servers. This blog post details some of the things I learned along the way.

Since the setup of Couch is a bit involved, I will divide this up over two blog

Provisioning Rookeries with Ansible

One of the stated goals of Rookeries is create a developer-friendly
blogging platform that is easier to install and setup than WordPress.
That is a tall order for a Python WSGI app, since there is some more
setup involved than just installing Apache and mod_php and unzipping
Wordpress into a folder. (Even with WordPress there is more involved
when doing a proper and maintainable setup.)

So while putting up a production ready Python WSGI app is more involved
technically, this does not mean the end-user needs to experience this.
That is where the Rookeries Ansible
comes into
play. I created that Ansible role to encapsulate the complexity of the
installing Rookeries. (This role uses [the nginx-uwsgi-supervisord Ansible
role which I wrote to handle the actual setup of a WSGI app on an bare-bones
Ubuntu server]
( All of the
details concerning the setup and configuration of a CouchDB server for a
Rookeries installation is included in the Rookeries Ansible role.

Installing Latest CouchDB on Ubuntu Linux

I use the latest Ubuntu LTS (14.04) for both my development and
deployment environments. Having the same environment reduces the effort for meI
to take Rookeries from development to production. However the
latest version of CouchDB for Ubuntu 14.04 is 1.5.0 and I wanted to use
the latest stable version of CouchDB. While upgrading between CouchDB
versions is straightforward, I know that I am less likely to upgrade to the
latest version of CouchDB once Rookeries stabilizes. And there is no
point on starting off with an older version of your database right from
the start of a project.

Fortunately the CouchDB devs distribute the latest stable version of
CouchDB via a convenient
. The
instructions on how to install CouchDB via the PPA is right on the
Launchpad page.

Installing via Console

# add the ppa
sudo add-apt-repository ppa:couchdb/stable -y
# update cached list of packages    
sudo aptitude update -y
# remove any existing couchdb binaries
sudo aptitude remove couchdb couchdb-bin couchdb-common -yf
# install the latest
sudo aptitude install couchdb

Provisioning via Ansible

The Rookeries Ansible role translates those instructions (minus the
removal of existing packages) to:

- name: add the couchdb ppa repository
  apt_repository: repo="ppa:couchdb/stable" state=present

- name: install couchdb
  apt: pkg=couchdb state=present
    - couchdb
    - couchdb-bin
    - couchdb-common

Running CouchDB

Now that we have CouchDB installed, we need to control it like we would any
other service on Linux server. Surprisingly enough when I tried to find the
packaged CouchDB service scripts (using the service command), I did not find

> sudo service --status-all
# ... A lot of entries but no couchdb ...

Turns out that CouchDB package comes with an Upstart script rather than
a traditional System V initrc script. (That itself is probably not a bad

> sudo status couchdb
couchdb start/running, process 5311
# There it is.

Starting and stopping service through Upstart is done via the ‘start’ and
‘stop’ commands. There are also ‘reload’ and ‘restart’ commands.

> sudo restart couchdb
couchdb start/running, process 15987

Side Note About Upstart vs Services vs Systemd

Update: I found an article that explains the evolution and the current situation of Linux service management. It explains things much better than I do and in much more detail. I learn quite a bit from it.

If you follow Linux developments and news, you might have heard about the development and controversy around new init systems. I will try to explain \nthese developments briefly here since we are on the topic of service scripts.

The old System V style for service scripts (in /etc/init.d/ or\n/etc/rc.d/) is not flexible when it comes to managing dependencies and running outside of the prescribed run-levels that happen during boot and shutdown.
However there is disagreement about what would would be a better alternative. Upstart was Canonical/Ubuntu’s attempt to create a more flexible system for managing services. However Debian and many other Linux distributions have recently switched over to another such system called systemd. Part of the controversy about systemd stems from the architectural design of systemd (which seems monolithic at first glance as it tries to solve service management, logging and few other seemly unrelated system level issues).

Another part of the controversy stems from how the project lead’s handled his previous project: PulseAudio. I will admit that my first experiences with PulseAudio were pretty rocky, and I missed how well using plain old ALSA worked. However these issues have since gone away, and I can not think of any PulseAudio or any audio issues I’ve encountered in Linux recently. (Ironically Windows 7 gives me more grief with sounds issues than Linux nowadays.)

I personally don’t know enough about systemd to form an opinion. Sure I am a bit anxious to see how this all plays out. However this is a case of wait and see. In the meantime be aware that the exact semantics on how you interact with services will change in the near future.

Update #2: An interview with Lennart Poettering about systemd, its design and intentions

Provisioning with Ansible

Fortunately Ansible does not make a distinction of what the underlying
service script setup is used. The Ansible service module works with initrc,
service, Upstart and systemd services without complaint.

In the Rookeries Ansible restarting the CouchDB service becomes a single

- name: stop couchdb server
  service: name=couchdb state=restarted

Next Up

In the next blog post I’ll write up about configuring and securing

Using CouchDB in Rookeries – Part 1 – Creating CouchDB Test Fixtures Using Bulk Updates

Back Story

I’ve been working on adding database persistence support to Rookeries. Instead of writing down my findings and losing them somewhere, I plan on documenting my findings and thoughts in a series of blog posts.

In the case of Rookeries that means connecting to and storing all of the journal, blog and page content as CouchDB documents. Since I want to implement this properly, I intend on adding tests to make sure I can manage CouchDB documents and databases properly. Rather than writing a number of tests that mock out CouchDB, I want to use a test database along with known test data fixtures for my tests.

Python CouchDB Integration for Rookeries

When looking at different CouchDB-Python binding libraries for Rookeries, I settled on py-couchdb. Manipulating CouchDB essentially means communicating with its REST API, so it is important that a Python binding library uses the sane approach to communicate with HTTP REST API. Unfortunately the more popular CouchDB-Python library uses only Python standard library and implements its HTTP mechanism in using standard library’s unintuitive modules. In contrast py-couchdb uses requests for querying the CouchDB server, making it a much more maintainable library.

Also py-couch offers Python query views, which I very much enjoy using at work. I still need to verify how well the library’s Python query server works in practise, but I will write a future blog post about my findings. py-couchdb lacks CouchDB-Python’s mapping functionality, which behaves similar to sqlalchemy’s ORM. However I am still debating on how I want to map between CouchDB documents and Pythonic domain objects.

Creating and Deleting CouchDB

Creating and deleting a database in a CouchDB server amounts to issuing a HTTP PUT or DELETE request against the server. This REST API provides no safety net nor confirmation about deleting a database, so one needs to be careful. py-couchdb provides a nice and simple API to create or delete a database as well.

Using cURL

# Create a CouchDB database
curl -X PUT http://admin:password@localhost:5984/my_database/

# DELETE a CouchDB database
curl -X DELETE http://admin:password@localhost:5984/my_database/

Using py-couchdb

# Create a CouchDB database
server = pycouchdb.client.Server('http://admin:password@localhost:5984')

# DELETE a CouchDB database

Inserting Fixture Data

Now that I can create a temporary test database, I need to populate it with some test data. Fortunately it turns out that CouchDB has a neat and fast way to insert data in bulk using its _bulk_docs API. With this API can easily come up with a number of documents that I want to input as test data.

Fixture Data Format

The format for inserting a mass of documents is:

  "docs": [
    {"_id": "1", "a_key": "a_value", "b_key": [1, 2, 3]},
    {"_id": "2", "a_key": "_random", "b_key": [5, 6, 7]},
    {"_id": "5", "a_key": "__etc__", "b_key": [1, 5, 5]}

Note that adding a _id specifies the CouchDB ID for the document.

Using cURL

# Bulk doc insert/update using the JSON data file.  One can also do this manually with a string.
curl -d @sample_data.json -X POST -H 'Content-Type: application/json' \

Using py-couchdb

UPDATED: 2015-Aug-22 I was totally wrong about the format of doing bulk updates to py-couchdb. Rather than the JSON format needed for CURL, a simple list of Python dictionaries works with the save_bulk() method. I’ve updated the code example.

import io
import json

# Best practice for writing unified Python 2 and 3 compatible code is 
# to use as a context manager. 
with'sample_data.json') as json_file:
    my_docs = json.load(json_file)
database = server.database('my_database')
# See my update note above, about the format save_bulk expects.


And with that, I have what I need to have repeatable tests. Hopefully this will land in Rookeries in the next couple of days.

Other Resources