Writing Integration Tests in Rust + Releasing Rookeries v0.11.0

As part of my overall change over in Rookeries, from Python to Rust, I rewrote a suite of integration tests for the server API. To celebrate my successful transition, I released version 0.11.0 of Rookeries, whose tests use pure Rust now!

I found the rewrite not too cumbersome, thanks to the wonderful guide in the Rust book on integration tests. I did miss pytest’s fixture setup, which makes testing really easy. Especially when setting up fixtures that are run only once per test suite. That said, a single session setup makes it difficult to run tests in parallel. And while I ran into some inconsistent tests, I had the exact same kind of problems in Python. Basically my data model for Rookeries, doesn’t work well for singleton data like a single site. In the future, I plan on having a single Rookeries instance managing multiple different sites, so this point of not having a single test setup is ultimately moot.

Building a common setup module turned out to be more work than I imagined. I could not use macros because of the package setup. Also since tests are compiled individually, some methods in the common module, simply are not called, leading to warning of dead code.

What made the transition is easy was using the reqwest crate, and heavy use of serde_json’s json! macro. Comparing the Python and Rust version of the tests, makes them look very comparable.

Rust Version src:

#[test]
fn test_authenticated_user_cannot_modify_site_using_bad_request() {
    test_site();
    let api_base_uri = common::api_base_url();
    let client = Client::new();
    let auth_token = auth_token_headers();
    let response = client
        .put(api_base_uri.join("/api/site").unwrap())
        .body("")
        .header(Authorization(auth_token))
        .send()
        .unwrap();

    assert_bad_request_response(response);
}

fn assert_bad_request_response(mut response: Response) {
    let expected_response_json = json!({
        "error": {
            "status_code": 400,
            "message": "Bad request",
        }
    });

    assert_eq!(response.status(), StatusCode::BadRequest);
    let actual_json_response = response.json::<Value>().unwrap();
    assert_eq!(actual_json_response, expected_response_json);
}

Python Version src:

def test_authenticated_user_cannot_modify_site_using_bad_request(
        auth_token_headers, api_base_uri, test_site):
    response = requests.put(
        url=f'{api_base_uri}/api/site',
        data='',
        headers=auth_token_headers,
    )
    assert_bad_request_response(response)

def assert_bad_request_response(response):
    expected_response_json = {
        'error': {
            'status_code': http.HTTPStatus.BAD_REQUEST.value,
            'message': mock.ANY,
        }
    }

    assert response.json() == expected_response_json
    assert response.status_code == http.HTTPStatus.BAD_REQUEST

I am pretty happy with how everything turned out overall. My next bit of Rookeries work will be migrating away from invoke and make to cargo make.

Adding Functional (End-to-End) Test to Rookeries

Testing the client side of Rookeries, has proven to be quite a challenge. Not necessarily because testing well-written React JS components is hard. Rather I found it hard to setup a proper and consistent unit test infrastructure to do so. Rather than going through the pain of writing and maintaining functional tests in Javascript, I decided to take a different path.

BDD + Web Testing – Theory and Practise

I wanted to write my tests in a business-domain-driven (BDD) style. While most developers find awkward to use at first, it is a great way to write out the business functionality and features of an app. It also forces you to think about your app in a non-technical manner, and prove to yourself (and others) that it does what you claim it does.

At work my team has been burned by slow and awkward web tests. Namely we worked with the Robot Framework, which uses its own DSL and is very difficult to work just as we wanted. Debugging tests was also quite unpleasant.http://lettuce.it/

Fortunately one of our newest team mates Kevin Qiu introduced us to lettuce (a BDD framework) and splinter (a Pythonic Selenium interface). And we’ve had a fair bit of success describing scenario using these tests. I won’t lie, Selenium is always temperamental. However the current batch of web tests have been very stable, and has very much convinced me that this setup can work effectively.

The Rookeries Take on BDD-style Web Testing

Rookeries usea a similar setup to what we’ve done at work: namely I use splinter to interface with Selenium. However unlike work, since Rookeries uses pytest instead of nose, I ended up using pytest-bdd to provide the BDD framework for Rookeries. Furthermore, I am using the pytest integration for splinter, to provide some of the browser fixtures needed for the tests.

Feel free to check out the functional tests in Rookeries to see examples of the tests. I also highly recommend watching Dylan Lacey ‘s talk about using Splinter for web testing at PyCon Australia 2013.

A Few Wrinkles I Found So Far

  • Dylan suggests using names for inputs when working with web tests. Unfortunately the react-bootstrap components I rely don’t include support for names, something I plan on submitting a patch/pull request for.
  • One needs to run a localized server as part of the functional tests, which makes for a slightly complicated task setup. This is something I need to simplify.
  • I found using the element.text of a container React component works around the issues of text phrases being broken up over a few components.
  • Using ids is the simplest way to find elements, even thought that isn’t what a user would actually use to navigate the site.
  • ipdb does not work very well when debugging tests. Plain old pdb works wonders though. I am considering switching over to using pdb++

Using CouchDB in Rookeries – Part 1 – Creating CouchDB Test Fixtures Using Bulk Updates

Back Story

I’ve been working on adding database persistence support to Rookeries. Instead of writing down my findings and losing them somewhere, I plan on documenting my findings and thoughts in a series of blog posts.

In the case of Rookeries that means connecting to and storing all of the journal, blog and page content as CouchDB documents. Since I want to implement this properly, I intend on adding tests to make sure I can manage CouchDB documents and databases properly. Rather than writing a number of tests that mock out CouchDB, I want to use a test database along with known test data fixtures for my tests.

Python CouchDB Integration for Rookeries

When looking at different CouchDB-Python binding libraries for Rookeries, I settled on py-couchdb. Manipulating CouchDB essentially means communicating with its REST API, so it is important that a Python binding library uses the sane approach to communicate with HTTP REST API. Unfortunately the more popular CouchDB-Python library uses only Python standard library and implements its HTTP mechanism in using standard library’s unintuitive modules. In contrast py-couchdb uses requests for querying the CouchDB server, making it a much more maintainable library.

Also py-couch offers Python query views, which I very much enjoy using at work. I still need to verify how well the library’s Python query server works in practise, but I will write a future blog post about my findings. py-couchdb lacks CouchDB-Python’s mapping functionality, which behaves similar to sqlalchemy’s ORM. However I am still debating on how I want to map between CouchDB documents and Pythonic domain objects.

Creating and Deleting CouchDB

Creating and deleting a database in a CouchDB server amounts to issuing a HTTP PUT or DELETE request against the server. This REST API provides no safety net nor confirmation about deleting a database, so one needs to be careful. py-couchdb provides a nice and simple API to create or delete a database as well.

Using cURL

# Create a CouchDB database
curl -X PUT http://admin:password@localhost:5984/my_database/

# DELETE a CouchDB database
curl -X DELETE http://admin:password@localhost:5984/my_database/

Using py-couchdb

# Create a CouchDB database
server = pycouchdb.client.Server('http://admin:password@localhost:5984')
server.create('my_database')

# DELETE a CouchDB database
server.delete('my_database')

Inserting Fixture Data

Now that I can create a temporary test database, I need to populate it with some test data. Fortunately it turns out that CouchDB has a neat and fast way to insert data in bulk using its _bulk_docs API. With this API can easily come up with a number of documents that I want to input as test data.

Fixture Data Format

The format for inserting a mass of documents is:

{
  "docs": [
    {"_id": "1", "a_key": "a_value", "b_key": [1, 2, 3]},
    {"_id": "2", "a_key": "_random", "b_key": [5, 6, 7]},
    {"_id": "5", "a_key": "__etc__", "b_key": [1, 5, 5]}
 ]
}

Note that adding a _id specifies the CouchDB ID for the document.

Using cURL

# Bulk doc insert/update using the JSON data file.  One can also do this manually with a string.
curl -d @sample_data.json -X POST -H 'Content-Type: application/json' \
   http://admin:password@localhost:5984/my_database/_bulk_docs

Using py-couchdb

UPDATED: 2015-Aug-22 I was totally wrong about the format of doing bulk updates to py-couchdb. Rather than the JSON format needed for CURL, a simple list of Python dictionaries works with the save_bulk() method. I’ve updated the code example.

import io
import json

# Best practice for writing unified Python 2 and 3 compatible code is 
# to use io.open as a context manager. 
with io.open('sample_data.json') as json_file:
    my_docs = json.load(json_file)
database = server.database('my_database')
# See my update note above, about the format save_bulk expects.
database.save_bulk(my_docs['docs'])

Conclusion

And with that, I have what I need to have repeatable tests. Hopefully this will land in Rookeries in the next couple of days.

Other Resources

The Seven Sins of Software “Engineering”

On Monday, my class discussed about using software “engineer”for software writers. I naturally argued against it. My mother finished civil engineering, and I strongly believe in the elegance, timelessness and excellence of engineering. This morning, as I struggled with and later killed my groupware program over a dispute of syncing with my Palm, I solidified my stance against the idea of treating programmers as engineers. Here are seven reasons why programmers are NOT engineers.

Sin 1: Documentation
Most programmers do not document their code. Some of them say that code is self-evident, self-describing and self-documenting. Tell that to the poor souls trying to extend or maintain that glob of code you brain-dumped.

Now take a look at a blueprint of an engineer. What would happen if some of the measurements disappeared or someone forgot to label the room that holds the AC.

Sin 2: Testing
Programmers do not test their programs effectively. You wonder why your state-of-the-art PC keeps on crashing, as soon as you plugin that expensive computer card? Or just randomly? Testing frameworks, and concepts exist, but with the exception of a few individuals, no one does a decent job of testing. Whining about different configurations and different users does not help.

When an engineer designs a skyscraper, he/she needs with the whims of people, animals and nature. That is why after an earthquake, skyscrapers and bridges still stand in San Francisco.

Sin 3: Research
Programmers rarely research anything. They simply build a new toy program, and expect everyone to love it. They expect that they can experiment away, and deploy an untested and unthoughtout innovation.
Then surprise and dismay occurs when their innovation is unwanted and even hated.

In engineering, most innovations need a huge amount of research. Engineers need to design, test, and re-test ideas until they know they work. An engineering innovation never builds upon a brand new untested technology.

Sin 4: Meeting Requirements
“I’m the programmer. I know what my client really wants.” Wrong! This cocky behaviour of coding what the programmer wants, and not what is required happens way too often. User requirements often go unresearched or ignored. The users become upset at the developers. Company gets into trouble. And the programmers are forced to fix their mistakes.

Engineering anything revolves around meeting some need or requirement. These requirements and the challenge of meeting is at the heart of engineering. If the client wants a skyscraper in the form of a glass inverted cone, the engineer has to build it. But the engineer must find out, understand and then build according to the clients and the universe’s requirements.

Sin 5: People Skills
When things go horribly wrong, who gets the blame? If a building falls (without an obvious catastrophic uncontrollable reason), the engineer gets the blame. If a user’s data vanishes in a simple operation, the user gets blamed. Programmers always blame their clients of incompetence, even when a problem is caused by bad programming.

This behaviour goes beyond just blaming. Programmers ignore the advice and problems of their users, and keep their jobs. An engineer who does likewise, gets fired.

Sin 6: Reliability
Programmers suffer from terminal featuritis-o-philia. They program a new feature and continue on innovating, all when their program lacks stability of any form. User start believing that programs and computers have a mind of their own, with data disappearing, malware crippling systems and hardware crashing inexplicably.

Engineers strive for reliability. Ever building is designed to last the harassment of humans, nature and even “acts of God”. Every vehicle should not kill its occupant, unless a total, catastrophic disaster occurs. Wondering why to this day, the crash-proof operating system and program is still a dream? Reliability takes second step to “innovation” and “coolness” in the minds of developers.

Sin 7: Overall Cuteness
If there is a single profession I can think of when you can get away with unprofessional “coolness”, it is software development. Programmers love to use obscure shorthands, bizarre implementations and wacky ideas, even competing annually for the “cutest” piece of code.

Cuteness is a sign of un-professionalism, and often immaturity in the real world. A cute engineer is a starving engineer. A cute programmer is a thriving one. See why programmer’s can not become software “engineers”?

Finale
My rant ends now, and I know many will complain that I overexaggerated. True, there are some programmers that do not sin in the above ways. They are the exception. Most programmers pass themselves off as good, and still sin terribly along the way. An engineer who sins as such is a bad engineer, and rightly so. I claim that if programmers ever want to earn the distinction of software engineers, they need to raise their standards considerably higher.