Rookeries v0.16.0 – Cross-Platform Support

On Wednesday night, I released version 0.16.0 of Rookeries, my developer/designer friendly static site generator! What makes this release exciting is that I added cross-platform support for macOS and FreeBSD! Also I setup an cross-platform installer, that works on all 64-bit x86 Linux distros including Windows Subsystem for Linux (WSL) 2.0. And I plan on supporting Linux, macOS (OS X) and FreeBSD as first-class systems going forward.

Also a multitude of stability and UX improvements landed in this release.
You can download Rookeries and install it using the new cross-platform installer here.

Rookeries v0.16.0 screenshot

Now this took quite some time and effort, especially around figuring out cross-compiling and what each OS could reasonably support. Over the course of the next few weeks, I will be writing about the challenges and how I overcame them when adding cross-platform support. (Including a quick aside why I dropped native non-WSL Windows support, and took up FreeBSD instead.)

Do you know that my mailing list is the first to know about releases and developments around Rookeries? Consider subscribing to the mailing list to be among the first to know!

Rookeries v0.13.1 – Release + Web Assembly Experiments

This past month I’ve been trying to determine the future direction I want to take my projects: especially Rookeries. Ultimately I want to build something that lasts and is able to sustain itself financially. As someone who is still a novice in the arts of marketing, and business development, this is hard, and doing this in an open, transparent environment like with libre software, just adds to it. While I have many musings on what is needed to run a sustainable open source project, they are just that: musings. Without getting into the weeds, I think the Ghost blogging platform team is doing the right thing. I recommend the Indie Hacker podcast that interviews John O’Nolan of Ghost.

Also I’ve been playing around with some of the interesting Rust Web Assembly projects out there. I think they have potential to improve Rookeries in a radical manner. However before I get anyone’s hopes up, I want to get something working before I announce the future direction of Rookeries. (Something, something, Web Assembly, something, something, plugin extensibility.) In the meantime I am releasing version 0.13.1 of Rookeries!

This version is mostly bug fixes, and code re-organization that will help me extend the project into the future.

P.S. My mailing list knows about my releases and plans of Rookeries, before the blog. So you should totally sign up for the Rookeries mailing list.

Rust + Gtk = Wow

As I’ve been experimenting with writing Rust apps, I attempted to create a small little GUI application. At first I attempted to setup everything with Qt, but C++/Qt interoperability with Rust is painful. Very, very painful. I experimented with some more radical UI frameworks such as Azul and Conrad. These have a lot of promise going forward. However for the here and now, I recommend looking at gtk-rs, Rust binding for Gtk.

With a bit of experimentation, looking through gtk-rs examples, some other projects using gtk-rs, and lots and lots of searching, I was able to create this:

This is a code viewer that lets you open Rust code, and view it. Yes, you are looking at a portion of the code that runs that code viewer. Getting the GtkSourceView working took some coaxing. Also I had to learn how to use the Glade UI editor for Gtk. Overall it took me about 2 to 3 hours to pull this off. I am very impressed with the results, and it opens up new possibilities for me.

Onward with Postgres and Diesel – Rookeries v0.13.0

That took longer than expected… but I’ve finally migrated Rookeries to use PostgreSQL 10! I finished the majority of the raw migration work about a week before PyCon Canada. However I did not want to confuse my updates and tweets related to PyCon Canada with my work on Rookeries. Also I wanted to add a few bits of polish, create a mailing list for Rookeries and fix up any issues I found with the site. (Quite a few data and TLS certificate issues happened along with this release.)

I found that migrating Rookeries from CouchDB to PostgreSQL quite challenging. This adds yet another data point to those that argue that fixing data models and databases can be significantly harder than fixing applications. One of the challenges I stumbled in was mapping the various keys in a CouchDB document for a resource, into a table. Fortunately the JSONB columns allowed for the flexibility to let me resolve some of my earlier problematic data designs later on. I also needed to familiarize myself with diesel.rs, the amazing ORM/database engine library for Rust. I struggled with understanding how custom types worked in Rust, and why they make database mapping of columns easy for more advanced mappings (like optionally nullable, JSONB and Array). However I managed to figure out even the hard parts using a combination of the documentation and by peeking into the test cases for diesel. Having a migration tool in the companion diesel-cli utility, provides the nicest UX for building out database migrations and mapping them to code. Overall the effort paid off and I am convinced that Rust has one of the cleanest and maintainable ORM/database libraries I’ve seen.

As mentioned earlier, I created a mailing list dedicated to Rookeries updates. I will continue to blog about releases, but if you are interested in the internals and development of Rookeries, then this mailing list a good thing to follow. Consider it like a journal that you can also reply back to and enter in a conversation about the development of Rookeries. I added a panel to the side of the site, that makes subscribing to Rookeries easy.

This will hopefully will be the last of the major technological shifts for the project, at least on the backend. The next releases I plan on concentrating on getting Rookeries in a state where I can drive Gatsby powered static sites via REST APIs. Once that becomes a reality, I will work on replacing my WordPress sites with Rookeries powered sites.

To Make or Not to Make – Using cargo make for Rookeries v0.12.0

I was pleasantly surprised when my last blog post about migrating to Rust’s integration tests really took off on Twitter. I did not quite expect that much interest. 🙂

Using cargo-make

I recently continued with my exploration of Rust through Rookeries (my attempt at a static site generator/backing API server). This time I worked on switching over from using invoke and GNU make to using a nice build system called cargo-make. Overall I am quite happy with the result. To give you a taste of how cargo-make describes build tasks and environment variables, I included a short snippet from the Rookeries Makefile.toml below:

<br />
[env]
SERVER_PORT = "5000"

[tasks.build-js]
description = "Build JS frontend."
command = "npm"
args = ["run", "build"]

[tasks.run-dev-server]
description = "Run a Rookeries dev server."
command = "cargo"
args = ["run", "--", "run", "${SERVER_PORT}"]
dependencies = ["build-js"]

Much like invoke and Make, you can specify dependencies. Unlike Make, there is no weird tab-based syntax, or implicit behaviour that requires a look up in the GNU Make manual. Just a simple TOML file. Environment variables can be specified in standalone .env files, passed in or specified in the Makefile.toml itself, and allow for variables specified during a run to override defaults, etc. cargo-make also lets you use conditional logic but I did not need to use that for my purposes.

For Rookeries, I ended up creating locally running tasks, and versions for the Dockerized build setup that I have in CircleCI. Also I ended up using the scripting support that cargo-make provides for preparing the database for, and running the integration tests.

My Thoughts on CouchDB for Rookeries and Rust

I feel I cheated a bit towards the end especially by using a few curl commands to instantiate the CouchDB database, and with running the API integration tests. Maybe I should of wrote a proper CouchDB library and utility. However I decided against it, simply because I want move away from CouchDB for Rookeries. So I may not want to invest that much time in maintaining a proper database client if I’ll end up switching over to using Postgresql via diesel.rs Originally, I started using CouchDB because I use it heavily at my work at Points. As time goes on, I find myself more and more reluctant with using CouchDB, just because of all the issues I’ve experienced at work. That said, if someone wants me to maintain a CouchDB Rust library/CLI based on the code I have for Rookeries, then please reach out to me. The CouchDB crates I saw in crates.io did not inspire me, so I wrote my own CouchDB module in Rust. But I don’t want to maintain that code just for myself, so if anyone actually needs a proper CouchDB crate and wants me to maintain it, please reach out to me.

Update – Sofa Library

After I wrote this blog post originally, I did not know about this nice CouchDB crate called sofa. If I had know about it, I would of not bothered with my own implementation.

New Release of Rookeries – Now 100% Rust

Finally since this migration was a bit of work, so I decided to release a new version of Rookeries. Since the invoke setup was the last Python code, Rookeries is now 100% completely written in Rust. (And the JS frontend naturally.) The Rookeries Docker image is available on Docker Hub. The project is still quite in flux, and what I will work on next depends on what I need to do to hook up Rookeries to one of my Gatsby powered sites.

Writing Integration Tests in Rust + Releasing Rookeries v0.11.0

As part of my overall change over in Rookeries, from Python to Rust, I rewrote a suite of integration tests for the server API. To celebrate my successful transition, I released version 0.11.0 of Rookeries, whose tests use pure Rust now!

I found the rewrite not too cumbersome, thanks to the wonderful guide in the Rust book on integration tests. I did miss pytest’s fixture setup, which makes testing really easy. Especially when setting up fixtures that are run only once per test suite. That said, a single session setup makes it difficult to run tests in parallel. And while I ran into some inconsistent tests, I had the exact same kind of problems in Python. Basically my data model for Rookeries, doesn’t work well for singleton data like a single site. In the future, I plan on having a single Rookeries instance managing multiple different sites, so this point of not having a single test setup is ultimately moot.

Building a common setup module turned out to be more work than I imagined. I could not use macros because of the package setup. Also since tests are compiled individually, some methods in the common module, simply are not called, leading to warning of dead code.

What made the transition is easy was using the reqwest crate, and heavy use of serde_json’s json! macro. Comparing the Python and Rust version of the tests, makes them look very comparable.

Rust Version src:

#[test]
fn test_authenticated_user_cannot_modify_site_using_bad_request() {
    test_site();
    let api_base_uri = common::api_base_url();
    let client = Client::new();
    let auth_token = auth_token_headers();
    let response = client
        .put(api_base_uri.join("/api/site").unwrap())
        .body("")
        .header(Authorization(auth_token))
        .send()
        .unwrap();

    assert_bad_request_response(response);
}

fn assert_bad_request_response(mut response: Response) {
    let expected_response_json = json!({
        "error": {
            "status_code": 400,
            "message": "Bad request",
        }
    });

    assert_eq!(response.status(), StatusCode::BadRequest);
    let actual_json_response = response.json::<Value>().unwrap();
    assert_eq!(actual_json_response, expected_response_json);
}

Python Version src:

def test_authenticated_user_cannot_modify_site_using_bad_request(
        auth_token_headers, api_base_uri, test_site):
    response = requests.put(
        url=f'{api_base_uri}/api/site',
        data='',
        headers=auth_token_headers,
    )
    assert_bad_request_response(response)

def assert_bad_request_response(response):
    expected_response_json = {
        'error': {
            'status_code': http.HTTPStatus.BAD_REQUEST.value,
            'message': mock.ANY,
        }
    }

    assert response.json() == expected_response_json
    assert response.status_code == http.HTTPStatus.BAD_REQUEST

I am pretty happy with how everything turned out overall. My next bit of Rookeries work will be migrating away from invoke and make to cargo make.

Rookeries v0.10.0 – Rust Re-write

I just rewrote Rookeries in Rust, and the latest version is now available as a Docker image on Docker Hub. (This is why I have not responded to emails in a bit… I’ve been doing a lot of thinking of what I want to do next.)

So why a rewrite? Ultimately I decided to change the direction of Rookeries as a project (making it more of a static site generation tool + headless CMS). I also wanted to improve my knowledge of Rust. (A programming language that I believe is quietly revolutionizing computing bit by bit. I really need to write about it sometime) But to show that there is reason for my madness:

Traditional CMS

I realized that my approach to Rookeries was heavily inspired by WordPress. And WordPress represents the best in class (in terms of ease-of-use) for traditional CMS. Traditional CMS being applications that control all the aspects of a website: pulling data that it organizes inside a database, the applying logic to manage the workflows of the data, and the creating and managing the user interface that ultimately shows the data. The traditional CMS has to do a lot of things. That usually means that the CMS developer needs to add a lot of assumptions and constraints to ship said CMS. And when those assumptions no longer hold, it takes a lot to change the codebase.

In the case of WordPress (and similar styled CMSs), the server has to do a lot of work with building a page. A WordPress site consists of both the site, and its admin console. While there has been work on updating the theme management and the page/post editor, ultimately there are limits to what kind of UIs that WordPress can support. WordPress also has a massive plugin marketplace, because the core developers do not know what every website will need. (Nor does it make sense to build everything in or every possibility.) For a modern WordPress powered site to stand-out, it needs needs a custom theme and often a half-dozen plugins. Naturally having so many moving parts, from different vendors (some theme and plugin devs are better than others) means there is a a greater chance of security issues. The routine upgrade of themes, plugins and the WordPress core is a rite that one has to diligently perform regularly.

It would be impossible for Rookeries to compete with WordPress head on, just given the number of hours I would need to sink to make things a reality. Also I do not think this desirable given the current direction.

The Alternative of Static Site Generation and API Driven Sites

The assumption that the server must handle most of the layout and display issues does not hold when dealing with modern browsers. Modern browsers can run rich multimedia experiences and real-time applications. Running intensive applications such as a 3D game is now possible, and will become come place. Ultimately the web is now a platform, a target that application developers can directly target.

As a result, many web agencies are turning to the JAMstack and having their websites act as their own full-blown applications. The applications either talk to a headless CMS acting an API or even foregoing that with static site generation. Probably the best example of this is Gatsby for generating sites built in React, which I am currently using to build out the Amber Penguin Software site. I also plan on eventually converting my other existing WordPress sites to use Gatsby or a similar tool. These tools as other web agencies found, separate out the complex logic of a frontend app from the backend and its data. These means more flexibility when building out a site. It also means that the backend CMS can be hardened and simplified to avoid many security issues. In fact with static site generation, the web app can be so removed from its CMS and database, that attacking a site becomes irrelevant. Such a site can also have better performance, and take advantage of caching and CDNs.

The Future of Rookeries

While there are a number of headless CMS and static site generators out there, I feel like the overall experience of working with them needs some improvement. There are no real good headless CMS systems written in Python or Rust, and I do think there are some really unpleasant rough edges around running a static site. Hence I plan on reworking Rookeries to help me build a CMS that can power my various sites. At the moment, this version 0.10.0 has feature parity with the previous version of Rookeries. I still need to rewrite some of the admin and testing in Rust, to avoid the need for Python. Also I need to figure out a nice manner to expose a flexible API to my sites. Fortunately I have enough examples to keep me busy for a bit.

Rookeries v0.9.0 out – New UI and Live Editing of Sites + Juggling JSON with jq Book Update

It has been quite a while since my last update. I apologize for the long silence. I wanted to focus on getting Rookeries up and running, to the degree that I can host a website. Namely I was hoping to update the Amber Penguin Software website and the Juggling JSON with jq book companion website to use Rookeries. Unfortunately I had to re-architecture the UI and bring the CouchDB management of the site and pages up to spec. However I am happy with the overall results:

Interactive Editing of Menus

Rookeries v0.9.0 introduces support for adding, removing and reordering menu items. It took a bit to find a good simple implementation of a drag and drop able list.

]4 Version 0.9.0 introduces interactive editing of menus.

Interactive Editing of Site Headers and Footers

Rookeries v0.9.0 also extends support of editing pages to site headers and footers. It uses the same inline editor for Markdown and HTML, that the pages currently enjoy.

]5 You can now edit site headers in Rookeries

So what else changed?

  • The migration of the UX over to a sliding sidebar is done.
  • Internally redux was replaced with MobX, which is far more maintainable for me.
  • The addition of saving sites and the current selected page.

There is still a lot to do on Rookeries, such as making the CMS more robust and flexible. Adding and removing pages is still outstanding. So is selection of pages via URLs. Marking whether or not changes have been made, to prompt the user to save. Also to check that the site is not in an invalid state. However things are starting to look up.

A few people have inquired about the state of the Juggling JSON with jq book, and when will a sample chapter and table of contents will be available. I apologize for not replying directly to those emails (I will do so shortly), but I hope to get those out in the next few days, and hopefully before the end of the year. Work on Rookeries has essentially ate up all the time that I would of used for working on the book. I think going forward I will work on both in parallel.

Adding Functional (End-to-End) Test to Rookeries

Testing the client side of Rookeries, has proven to be quite a challenge. Not necessarily because testing well-written React JS components is hard. Rather I found it hard to setup a proper and consistent unit test infrastructure to do so. Rather than going through the pain of writing and maintaining functional tests in Javascript, I decided to take a different path.

BDD + Web Testing – Theory and Practise

I wanted to write my tests in a business-domain-driven (BDD) style. While most developers find awkward to use at first, it is a great way to write out the business functionality and features of an app. It also forces you to think about your app in a non-technical manner, and prove to yourself (and others) that it does what you claim it does.

At work my team has been burned by slow and awkward web tests. Namely we worked with the Robot Framework, which uses its own DSL and is very difficult to work just as we wanted. Debugging tests was also quite unpleasant.http://lettuce.it/

Fortunately one of our newest team mates Kevin Qiu introduced us to lettuce (a BDD framework) and splinter (a Pythonic Selenium interface). And we’ve had a fair bit of success describing scenario using these tests. I won’t lie, Selenium is always temperamental. However the current batch of web tests have been very stable, and has very much convinced me that this setup can work effectively.

The Rookeries Take on BDD-style Web Testing

Rookeries usea a similar setup to what we’ve done at work: namely I use splinter to interface with Selenium. However unlike work, since Rookeries uses pytest instead of nose, I ended up using pytest-bdd to provide the BDD framework for Rookeries. Furthermore, I am using the pytest integration for splinter, to provide some of the browser fixtures needed for the tests.

Feel free to check out the functional tests in Rookeries to see examples of the tests. I also highly recommend watching Dylan Lacey ‘s talk about using Splinter for web testing at PyCon Australia 2013.

A Few Wrinkles I Found So Far

  • Dylan suggests using names for inputs when working with web tests. Unfortunately the react-bootstrap components I rely don’t include support for names, something I plan on submitting a patch/pull request for.
  • One needs to run a localized server as part of the functional tests, which makes for a slightly complicated task setup. This is something I need to simplify.
  • I found using the element.text of a container React component works around the issues of text phrases being broken up over a few components.
  • Using ids is the simplest way to find elements, even thought that isn’t what a user would actually use to navigate the site.
  • ipdb does not work very well when debugging tests. Plain old pdb works wonders though. I am considering switching over to using pdb++

Adding a Code Editor to Rookeries

One of the main selling points for Rookeries, is its ability to let the
writer write content using Markdown, and getting a live preview of the result.
Since the user will be writing in code, it would be a better experience to
use a text editor meant for code. Rather than try to undertake a large task
like this myself, I decided to look around for an editor.

Choosing a Text Editor

My three main concerns for a code editor is for it work properly, integrate
nicely with React and have nice syntax highlighting. After a bit of searching
I found out about the following code editors:

Setting up a Code Mirror-React Component

In the end I chose Code Mirror and specifically the react-codemirror project.
The syntax highlighting does not quite work yet, but I will work on that. The
other integration looked more mature, but ultimately did not want to work for
me.

Setting up the Code Mirror component in Coffee-JSX turned out fairly easy:

React = require 'react'
CodeMirror = require 'react-codemirror'

MyView = React.createClass(
    render: () ->
        options = {lineNumbers: true, mode:'markdown', theme: 'monokai'}
        return (
            <codemirror value={ code } onChange={ @updateCode } options={ options }></codemirror>
        )
)

CSS Gotcha!

Oddly enough just setting up the CodeMirror component did not work right out
of the box. In fact the result was rendering each line as a separated out

 box.  After a bit of digging, I found that the styling was based on the 
pre-existing CSS style provided by Bootstrap... not by Code Mirror!  

The reason turned out to be simple: I had not included the CSS from Code 
Mirror into the app. Since I did not find a CDN with Code Mirror setup, I 
decided to simply include Code Mirror's CSS into my LESS stylesheets for 
Rookeries, based on the example in react-codemirror:

    @import (inline) "../../../node_modules/codemirror/lib/codemirror.css";

This worked well enough, and I've included some of the CodeMirror themes as 
well.

# Outstanding Issues

Currently the text editor works however the setup is missing an important 
feature: syntax highlighting.  In theory Code Mirror should enable syntax 
highlighting based on the passed in mode.  Setting the mode to 'markdown' does 
not seems to achieve the expected result.