PyCon Canada 2015 Talk – Fabric-less Deployment of WSGI Apps

As you may know, I presented a talk at PyCon Canada this year. And as promised I wanted to post the video and the associated content here. Hope you enjoy it!

Video of Talk

Abstract

Intermediate level talk about migrating a Fabric deployment to a more flexible setup using using Ansible and Invoke. Follows the journey of changing the deployment of a Flask based blogging app from a Fabric script and pre-provisioned server to a modular system with Invoke tasks and provisioning using Ansible. Discusses the advantages and cons of moving to a declarative system versus direct shell commands. Touches upon on Ansible Roles, Ansible Galaxy and Invoke.

Adding Functional (End-to-End) Test to Rookeries

Testing the client side of Rookeries, has proven to be quite a challenge. Not necessarily because testing well-written React JS components is hard. Rather I found it hard to setup a proper and consistent unit test infrastructure to do so. Rather than going through the pain of writing and maintaining functional tests in Javascript, I decided to take a different path.

BDD + Web Testing – Theory and Practise

I wanted to write my tests in a business-domain-driven (BDD) style. While most developers find awkward to use at first, it is a great way to write out the business functionality and features of an app. It also forces you to think about your app in a non-technical manner, and prove to yourself (and others) that it does what you claim it does.

At work my team has been burned by slow and awkward web tests. Namely we worked with the Robot Framework, which uses its own DSL and is very difficult to work just as we wanted. Debugging tests was also quite unpleasant.http://lettuce.it/

Fortunately one of our newest team mates Kevin Qiu introduced us to lettuce (a BDD framework) and splinter (a Pythonic Selenium interface). And we’ve had a fair bit of success describing scenario using these tests. I won’t lie, Selenium is always temperamental. However the current batch of web tests have been very stable, and has very much convinced me that this setup can work effectively.

The Rookeries Take on BDD-style Web Testing

Rookeries usea a similar setup to what we’ve done at work: namely I use splinter to interface with Selenium. However unlike work, since Rookeries uses pytest instead of nose, I ended up using pytest-bdd to provide the BDD framework for Rookeries. Furthermore, I am using the pytest integration for splinter, to provide some of the browser fixtures needed for the tests.

Feel free to check out the functional tests in Rookeries to see examples of the tests. I also highly recommend watching Dylan Lacey ‘s talk about using Splinter for web testing at PyCon Australia 2013.

A Few Wrinkles I Found So Far

  • Dylan suggests using names for inputs when working with web tests. Unfortunately the react-bootstrap components I rely don’t include support for names, something I plan on submitting a patch/pull request for.
  • One needs to run a localized server as part of the functional tests, which makes for a slightly complicated task setup. This is something I need to simplify.
  • I found using the element.text of a container React component works around the issues of text phrases being broken up over a few components.
  • Using ids is the simplest way to find elements, even thought that isn’t what a user would actually use to navigate the site.
  • ipdb does not work very well when debugging tests. Plain old pdb works wonders though. I am considering switching over to using pdb++

Merging Git Repositories

One of the things that I realized while working on Rookeries, is that the server and client have to be more or less developed in sync. In theory one could develop these in parallel, however a function needs to be build on both sides. Since for the most part, I’m building Rookeries by myself having two separate repositories only gives me more things to maintain.

So I decided to merge the rookeries-react-app repository back into the rookeries repository. I used the instructions found in this post on merging git repositories to successfully achieve that.. In case this played out as:

git clone https://bitbucket.org/dorianpula/react-rookeries-app temp
cd temp
git filter-branch --prune-empty --tree-filter '
     if [[ ! -e client ]]; then 
    mkdir -p client 
    git ls-tree --name-only $GIT_COMMIT  | xargs -I files mv files client fi'

cd ..
git clone https://bitbucket.org/dorianpula/rookeries rookeries-next
cd rookeries-next
git remote add --fetch temp ../temp 
git merge -s ours --no-commit temp/master 
git read-tree --prefix=client/ -u temp/master 
git commit -m "Merged rookeries-react-app in rookeries" 
git pull -s subtree temp master

git push origin master

Remembrance Day

enter image description here

In Flanders fields the poppies blow
Between the crosses, row on row,
That mark our place; and in the sky
The larks, still bravely singing, fly
Scarce heard amid the guns below.

We are the Dead. Short days ago
We lived, felt dawn, saw sunset glow,
Loved and were loved, and now we lie
In Flanders fields.

Take up our quarrel with the foe:
To you from failing hands we throw
The torch; be yours to hold it high.
If ye break faith with us who die
We shall not sleep, though poppies grow
In Flanders fields.

Blog Break

Hey there Reader!

Over the past couple of weeks I have tried to post a new article or blog every week. And normally I would do the same today. Heck I do have a few posts in draft that I started working and I plan to publish a bit later.

However for the next couple of weeks I will be taking a break from blogging to concentrate on preparing for my PyCon Canada talk, getting some more traction on Rookeries and a new project that I plan on unveiling soon.

I’ll return to my regular blogging schedule sometime in mid or end of November once the dust settles.

Adding a Code Editor to Rookeries

One of the main selling points for Rookeries, is its ability to let the
writer write content using Markdown, and getting a live preview of the result.
Since the user will be writing in code, it would be a better experience to
use a text editor meant for code. Rather than try to undertake a large task
like this myself, I decided to look around for an editor.

Choosing a Text Editor

My three main concerns for a code editor is for it work properly, integrate
nicely with React and have nice syntax highlighting. After a bit of searching
I found out about the following code editors:

Setting up a Code Mirror-React Component

In the end I chose Code Mirror and specifically the react-codemirror project.
The syntax highlighting does not quite work yet, but I will work on that. The
other integration looked more mature, but ultimately did not want to work for
me.

Setting up the Code Mirror component in Coffee-JSX turned out fairly easy:

React = require 'react'
CodeMirror = require 'react-codemirror'

MyView = React.createClass(
    render: () ->
        options = {lineNumbers: true, mode:'markdown', theme: 'monokai'}
        return (
            <codemirror value={ code } onChange={ @updateCode } options={ options }></codemirror>
        )
)

CSS Gotcha!

Oddly enough just setting up the CodeMirror component did not work right out
of the box. In fact the result was rendering each line as a separated out

 box.  After a bit of digging, I found that the styling was based on the 
pre-existing CSS style provided by Bootstrap... not by Code Mirror!  

The reason turned out to be simple: I had not included the CSS from Code 
Mirror into the app. Since I did not find a CDN with Code Mirror setup, I 
decided to simply include Code Mirror's CSS into my LESS stylesheets for 
Rookeries, based on the example in react-codemirror:

    @import (inline) "../../../node_modules/codemirror/lib/codemirror.css";

This worked well enough, and I've included some of the CodeMirror themes as 
well.

# Outstanding Issues

Currently the text editor works however the setup is missing an important 
feature: syntax highlighting.  In theory Code Mirror should enable syntax 
highlighting based on the passed in mode.  Setting the mode to 'markdown' does 
not seems to achieve the expected result.

Giving a Talk at PyCon Canada 2015

I am super excited to announce that I will be doing a talk at PyCon Canada this year! I will be talking about migrating from using Fabric to deploy my WSGI app (Rookeries) to using a combination of Invoke and Ansible. PyCon Canada will be happening in Toronto at the University of Toronto campus Saturday November 7 to Sunday November 8, 2015. My talk will on the Sunday at 3:45-4:15 PM. Videos of the talks should be available about a day or two after the talk. I look forward to seeing everyone there!

More info on my talk.

Also I plan on being at the Sprints the following Monday as well.

Fixing SPDX Expression Warning in package.json

If you ever run into the following warning when installing your NPM package:

npm WARN package.json rookeries-api-client-wrapper@0.4.9 license should be a valid SPDX license expression

That means you have an improper a name for the specified license in your package.json. So what are valid values for licenses? Well… here is the SPDX list of licenses

Command-line JSON Formatting with jq

About 2 or 3 months ago, when testing a deployment of a microservice at work
with Eric, our head Ops admin, we were looking at the JSON output of one of
the REST endpoints. Rather than looking at the raw output from curl, I
piped the output through JSON tool in the Python standard library:

$ curl -X GET -s http://rookeries.org/status | python -m json.tool
{
    "app": "rookeries",
    "version": "0.4.9"
}

(I will give you a couple of examples based on calls to Rookeries, rather than
the actual service call, since that API isn’t available publicly yet.)

Eric seeing that suggested I try out a utility that he uses: jq which is
great for formatting and querying JSON
.

Installing jq

jq is a utility in C, and fortunately there are binary packages available for
it in the latest Ubuntu LTS (14.10). Installing it via aptitude (or apt-get):

$ sudo aptitude install jq

Pretty Printing JSON

The simplest use case for jq is to simply pretty print JSON output. This is
done by piping the result of a CURL command to jq with the parameter ‘.’:

$ curl -X GET -s http://status.bitbucket.org/api/v2/status.json | jq .
{
  "status": {
    "description": "All Systems Operational",
    "indicator": "none"
  },
  "page": {
    "updated_at": "2015-09-03T08:03:55.275Z",
    "url": "http://status.bitbucket.org",
    "name": "Atlassian Bitbucket",
    "id": "bqlf8qjztdtr"
  }
}

Formatting JSON in jq

It is also possible to format the output of the JSON to display only relevant
information. For instance if I want to find out the status of the components
that make up Bitbucket I can do the following:

$ curl -X GET -s http://status.bitbucket.org/api/v2/components.json | jq .

That however will give me a whole lot of extra data that I might not want. So
instead I might to narrow down and re-format the JSON data to something more
manageable with some jq magic:

$ curl -X GET -s http://status.bitbucket.org/api/v2/components.json | \ 
jq '{src: .page, components: [.components[] | \
    {id: .id, name: .name, status: .status}]}'

{
  "components": [
    {
      "status": "operational",
      "name": "Website",
      "id": "g0lfj4sv2fhf"
    },
    {
      "status": "operational",
      "name": "API",
      "id": "k0x2yw1435v7"
    },
    {
      "status": "operational",
      "name": "SSH",
      "id": "qmh4tj8h5kbn"
    },
    {
      "status": "operational",
      "name": "Git via HTTPS",
      "id": "c1qmcrcbc5zy"
    },
    {
      "status": "operational",
      "name": "Mercurial via HTTPS",
      "id": "vmbzxbbjz05j"
    },
    {
      "status": "operational",
      "name": "Webhooks",
      "id": "rfzky0v13fbp"
    },
    {
      "status": "operational",
      "name": "Source downloads",
      "id": "28h8dvv2qfzw"
    }
  ],
  "src": {
    "updated_at": "2015-09-03T08:03:55.275Z",
    "url": "http://status.bitbucket.org",
    "name": "Atlassian Bitbucket",
    "id": "bqlf8qjztdtr"
  }
}

I won’t explain that particular string in detail. But it will basically
craft a new JSON object, and generate new filtered objects when iterating
over the old array. Overall jq is pretty neat and is extremely fast to work
with.

There is also a Python bindings library for jq. I am considering using it to
help with mapping JSON into Python objects. However I have not played around
with it long enough to know if the extra dependencies are worthwhile or
whether or not it will bring a lot of benefits to Rookeries.

Reference Links

Using CouchDB in Rookeries – Part 3 – Configuring a Remote CouchDB Server

In the previous instalment of this series I wrote about installing and
managing on a remote server. Now lets talk about configuring CouchDB so that
it can run as a production server. This will not cover CouchDB’s
configuration extensively, rather I will touch on the parts relevant to
Rookeries.

Configuring CouchDB

CouchDB can be configured in two way: either by modifying the INI setting files
located in the /etc/couchdb/ or by visiting the Configuration UI in Futon:
e.g. http://localhost:5984/_utils/config.html I went with the route of editing the local.ini file via Ansible.

Configuring Users Authentication on CouchDB

The default installation of CouchDB does not force you to declare and secure
users. Users and user authentication is totally optional. However since I
did not want open up my production database to the world.

Adding an Admin User

I first added an admin user to the CouchDB configuration. This user being the
admin user for the entire CouchDB server, rather than an individual database.
The change consisted of adding a value for the user and password, under the
admins section:

[admins]
admin = password

If you’re worried about leaving your password in plain-text in the
configuration, then don’t. After restarting CouchDB (via the Upstart service)
this password gets hashed.

Admin Party No More – Enforcing User Authentication

By default CouchDB runs in what is “admin party” mode, meaning you do not need to log in to make admin or user changes on the server. Naturally for a production server that is not something you want to do. So you have to enable requiring user login:

[couch_httpd_auth]
require_valid_user = true

Getting CouchDB to Talk with the Rest of the World

This part is optional, however if you want CouchDB accessible from more than
the localhost you have configure it to allow connections from multiple sites.
I needed this since I use Codeship as a continuous integration (CI) service for Rookeries, and I wanted to run integration and end-to-end tests using my production database server. (In an ideal world I would have a separate CouchDB server just for testing or have a CI that has a local instance of CouchDB.)

Binding Addresses

The trick to allowing this is to set the right binding address for CouchDB.
This can be done by changing the bind_address value in the httpd section of the configuration as such:

[httpd]
bind_address = 0.0.0.0

By default this just localhost or 127.0.0.1. You can also setup the
configuration differently. One thing that I am not sure of is passing a
range or a list of different bind addresses. I am not sure this is possible
based on the documentation that I have seen.

What about HTTPS?

CouchDB has options to handle HTTPS and SSL natively. I personally have not configured my site to use HTTPS, since none of my sites
do so currently. Getting certificates and everything setup for all my sites
is a bit involved so I have avoided the issue for the time being. I plan on
getting around to do so in the future.

However if you have the time and option to setup HTTPS, please do so! Putting
up another layer of security around a production CouchDB will help. More
importantly HTTPS gives you and your end users a degree of privacy, that is
rare in these Post-Snowdown times.