Tag Archives: Rust

Rookeries v0.17.0 – Auto-Building + Multi-Template Support

Posted on by

Recovering from PyCon Canada 2019, and the holidays took longer than I expected. However I am pleased to announce the release of version 0.17.0 of Rookeries to start off this new year!

Version 0.17.0 includes a number of improvements to the serve command:

  • Auto-building of the site when a file in a project changes.
  • Opening of the resulting build site in a browser using the --open flag.
  • Auto-refreshing of the opened site when in development mode.

In addition, I added:

  • Multi-template support which will allow for building out complex templates in the future.
  • Improvements to the UX experience.

If you are running version 0.16.0, you can run a re-install using the rookeries-online-installer.sh to update to the latest version. Otherwise you can install v0.17.0 by following the instruction on the Download page.

Next Steps

As for what I plan to work on next, I want to improve the infrastructure around downloading, installing and working with both plugins and templates. I will be using my new site for my Juggling JSON with jq Cookbook ebook as a test bed for improving working with Rookeries. That way I have some real use cases outside of Rookeries’ project page. I am looking forward to working on both projects and see how they both develop in parallel.

Do you know that my mailing list is the first to know about releases and developments around Rookeries? Consider subscribing to the mailing list to be among the first to know!

Speaking At PyCon Canada 2019 – Rust Accelerated Pythons

Posted on by

I am happy to announce that I will be giving a talk on Rust and Python at PyCon Canada 2019: Rust Accelerated Pythons!


I will be talking about how to write Python bindings for Rust. With Rust you can create and call fast native code without worrying about having to use C/C++. And creating Python bindings is pretty easy in Rust. This talk will give an introduction to the Rust language, the challenges of writing cross-language bindings, and an example of working with the excellent PyO3 library. Hope to see you there if you are in the Toronto area for PyCon Canada!

If you can’t make it out to PyCon Canada this year, I also plan on giving this talk at an upcoming Rust Toronto meetup. More details will on that in a future post.

Rookeries v0.16.0 – Cross-Platform Support

Posted on by

On Wednesday night, I released version 0.16.0 of Rookeries, my developer/designer friendly static site generator! What makes this release exciting is that I added cross-platform support for macOS and FreeBSD! Also I setup an cross-platform installer, that works on all 64-bit x86 Linux distros including Windows Subsystem for Linux (WSL) 2.0. And I plan on supporting Linux, macOS (OS X) and FreeBSD as first-class systems going forward.

Also a multitude of stability and UX improvements landed in this release.
You can download Rookeries and install it using the new cross-platform installer here.

Rookeries v0.16.0 screenshot

Now this took quite some time and effort, especially around figuring out cross-compiling and what each OS could reasonably support. Over the course of the next few weeks, I will be writing about the challenges and how I overcame them when adding cross-platform support. (Including a quick aside why I dropped native non-WSL Windows support, and took up FreeBSD instead.)

Do you know that my mailing list is the first to know about releases and developments around Rookeries? Consider subscribing to the mailing list to be among the first to know!

Rust + Gtk = Wow

Posted on by

As I’ve been experimenting with writing Rust apps, I attempted to create a small little GUI application. At first I attempted to setup everything with Qt, but C++/Qt interoperability with Rust is painful. Very, very painful. I experimented with some more radical UI frameworks such as Azul and Conrad. These have a lot of promise going forward. However for the here and now, I recommend looking at gtk-rs, Rust binding for Gtk.

With a bit of experimentation, looking through gtk-rs examples, some other projects using gtk-rs, and lots and lots of searching, I was able to create this:

This is a code viewer that lets you open Rust code, and view it. Yes, you are looking at a portion of the code that runs that code viewer. Getting the GtkSourceView working took some coaxing. Also I had to learn how to use the Glade UI editor for Gtk. Overall it took me about 2 to 3 hours to pull this off. I am very impressed with the results, and it opens up new possibilities for me.

Packaging up a Rust Binary for Linux

Posted on by

Prologue

I should of written an update for quite some time. While I’ve been experimenting with marketing analytics, learning about data science, business development, doing DevOps with GitLab CI and various other things, I wanted to write up my learning when I had a chance to internalize everything. However what made me decide to write an update is this tweet from Chris Krycho. Chris runs the amazing New Rustacean podcast, which is a must listen for anyone interested in learning about programming in Rust.

How does one package a Rust app?

Chris asked about finding a good way to distribute Rust binaries across Linux distros:

Interestingly enough, I recently figured out how to package a small Rust CLI utility that I’m working on. My response was:

This post elaborates on what I meant with my reply.

Building snap packages using snapcraft

The fine folks at Canonical (the makers of Ubuntu Linux), have created something called snap packages. These packages and associated package manager help developers distribute applications (desktop, cloud, etc.) in a safe, isolated manner. I currently have slack installed this way. snaps isolate apps by having the package maintainer declare the capabilities an app requires (network access, access to system files, the GPU, home directory, etc.), and then ensuring the apps can not escape this sandbox.

Basic Setup

Getting setup with packaging a Rust crate was not too hard:

  1. Install snapcraft: sudo snap install snapcraft --classic
  2. Create a snap template inside your crate project: snap init
  3. Edit the generated snap/snapcraft.yml as per the documentation.
  4. Build the snap using snapcraft.
  5. Install the resulting snap with sudo snap install my-cool-rust-bin_x.y.z.snap --devmode --dangerous (this assuming you are experimenting with building a snap)
  6. Add you should be able to distribute your app as a snap now. (See the caveats below.)

Caveats working with snaps

Now there a bunch of caveats when working with snaps. And for my own Rust utility, I found these too taxing and I decided to go with creating a standard Debian package instead. However if I wanted to target multiple distributions and my app didn’t have a very unorthodox setup (my app relies on using the Chrome WebDriver to control a networked device managed by dd-wrt), I would probably have gone with a snap instead:

  • You need to know what capabilities your app needs: file access, network, etc. and you need to declare the appropriate interfaces in your snapcraft.yml
  • Using something other than my local system (be it a Docker based build or using a different base like base18), failed terribly at least for Rust.
  • Whether or not I’d have more success if my base system was the recommended Ubuntu 16.04 and not 18.04 is an outstanding mystery.
  • The snap confinement, even on the much more liberal devmode, works very well. No amount of coaxing on my part, let me use system paths when trying to spawn a process. This could just be me though, as not declaring network access did not block my app.
  • The docs could of been clearer about what was the latest recommended approach. (Still way clearer than the documentation for creating a DEB or RPM from scratch.)
  • Knowing which libraries (for the type of base system) your app needs takes a bit of experimentation. (e.g. I needed libssl1.0 for some builds and libssl1.1)
  • I have no idea how the classical confinement should work, and it is not recommended either way.

The end result for me was a working snap package, but an app that would not work when called from an installed snap. However I think snap packages are probably the way to go moving forward (or a similar format like flatpak). Since I only plan on targeting local Ubuntu 18.04 installs, I ended up creating a Debian package instead.

Building a Debian package with Cargo

I found a nice utility for creating a deb out of a Rust crate, called cargo-deb. After installing the crate with cargo: cargo install cargo-deb, I simply ran cargo deb and I was done. cargo-deb looked into my Cargo.toml for the metadata, ran a build and a few moments later I was the proud owner of a Debian package. Since my app relies on the chromium-browser and chromium-chromedriver packages, I added a small section in my Cargo.toml as so:

[package.metadata.deb]
depends = "$auto, chromium-browser, chromium-chromedriver"

The $auto is something that the Debian packaging mechanism needs, and that is the comma separated format that DEBs use.

Building a RPM from a DEB package

Now this the part that I didn’t do this time around. However I figured out how to create RPM packages from DEB packages a few months ago. The trick is to use the alien utility to create a RPM out of a DEB:

sudo alien --to-rpm --scripts --verbose my-cool-rust-bin.deb

For the record, I did not try to improve or debug the resulting RPM. (This entire effort was for a product that failed to launch.) However as part of my tests I was able install it and run it from on CentOS VM.

Epilogue

Anyways, I would recommend the cargo-deb and alien approach, if you are not planning to distribute a Rust app across a multitude of Linux distros. I would recommend dipping into snap if you plan on distributing something more commercial and wide-spread like a slack or kubectl. And I hope that helps you on your Rust app packaging for Linux journey!

Embedded Rust Library Experiment for Python and Web Assembly

Posted on by

With my ever growing list of things that I need to catch up (like wiring my home network and managing Rookeries), I needed a small fun project that I can work on. Ever since I learned enough Rust to be able to convert Rookeries, I wanted to play around with being able to speed up my code with a Rust library. I am especially interested in figuring how to call Rust code from Python or from JS with Web Assembly.

As a test bed (and a reason) for me to learn this, I created a small little library for getting the uptime of a local server (Linux only): embedded-uptime converting between different measurement units like Celsius and Fahrenheit: embedded-unit-converter. If you’d like to follow along, feel free to check it out. I will be posting updates on the blog, and on the Rookeries mailing list.

Updated on 2019 February 4: When I setup the project, I forgot that server uptimes that rely on accessing a server’s /proc/uptime can not possibly work in Web Assembly in browser environment. After some consideration I decided to go with something simple that accessible from any platform, namely conversion between different units of measure.

To Make or Not to Make – Using cargo make for Rookeries v0.12.0

Posted on by

I was pleasantly surprised when my last blog post about migrating to Rust’s integration tests really took off on Twitter. I did not quite expect that much interest. 🙂

Using cargo-make

I recently continued with my exploration of Rust through Rookeries (my attempt at a static site generator/backing API server). This time I worked on switching over from using invoke and GNU make to using a nice build system called cargo-make. Overall I am quite happy with the result. To give you a taste of how cargo-make describes build tasks and environment variables, I included a short snippet from the Rookeries Makefile.toml below:

<br />
[env]
SERVER_PORT = "5000"

[tasks.build-js]
description = "Build JS frontend."
command = "npm"
args = ["run", "build"]

[tasks.run-dev-server]
description = "Run a Rookeries dev server."
command = "cargo"
args = ["run", "--", "run", "${SERVER_PORT}"]
dependencies = ["build-js"]

Much like invoke and Make, you can specify dependencies. Unlike Make, there is no weird tab-based syntax, or implicit behaviour that requires a look up in the GNU Make manual. Just a simple TOML file. Environment variables can be specified in standalone .env files, passed in or specified in the Makefile.toml itself, and allow for variables specified during a run to override defaults, etc. cargo-make also lets you use conditional logic but I did not need to use that for my purposes.

For Rookeries, I ended up creating locally running tasks, and versions for the Dockerized build setup that I have in CircleCI. Also I ended up using the scripting support that cargo-make provides for preparing the database for, and running the integration tests.

My Thoughts on CouchDB for Rookeries and Rust

I feel I cheated a bit towards the end especially by using a few curl commands to instantiate the CouchDB database, and with running the API integration tests. Maybe I should of wrote a proper CouchDB library and utility. However I decided against it, simply because I want move away from CouchDB for Rookeries. So I may not want to invest that much time in maintaining a proper database client if I’ll end up switching over to using Postgresql via diesel.rs Originally, I started using CouchDB because I use it heavily at my work at Points. As time goes on, I find myself more and more reluctant with using CouchDB, just because of all the issues I’ve experienced at work. That said, if someone wants me to maintain a CouchDB Rust library/CLI based on the code I have for Rookeries, then please reach out to me. The CouchDB crates I saw in crates.io did not inspire me, so I wrote my own CouchDB module in Rust. But I don’t want to maintain that code just for myself, so if anyone actually needs a proper CouchDB crate and wants me to maintain it, please reach out to me.

Update – Sofa Library

After I wrote this blog post originally, I did not know about this nice CouchDB crate called sofa. If I had know about it, I would of not bothered with my own implementation.

New Release of Rookeries – Now 100% Rust

Finally since this migration was a bit of work, so I decided to release a new version of Rookeries. Since the invoke setup was the last Python code, Rookeries is now 100% completely written in Rust. (And the JS frontend naturally.) The Rookeries Docker image is available on Docker Hub. The project is still quite in flux, and what I will work on next depends on what I need to do to hook up Rookeries to one of my Gatsby powered sites.

Writing Integration Tests in Rust + Releasing Rookeries v0.11.0

Posted on by

As part of my overall change over in Rookeries, from Python to Rust, I rewrote a suite of integration tests for the server API. To celebrate my successful transition, I released version 0.11.0 of Rookeries, whose tests use pure Rust now!

I found the rewrite not too cumbersome, thanks to the wonderful guide in the Rust book on integration tests. I did miss pytest’s fixture setup, which makes testing really easy. Especially when setting up fixtures that are run only once per test suite. That said, a single session setup makes it difficult to run tests in parallel. And while I ran into some inconsistent tests, I had the exact same kind of problems in Python. Basically my data model for Rookeries, doesn’t work well for singleton data like a single site. In the future, I plan on having a single Rookeries instance managing multiple different sites, so this point of not having a single test setup is ultimately moot.

Building a common setup module turned out to be more work than I imagined. I could not use macros because of the package setup. Also since tests are compiled individually, some methods in the common module, simply are not called, leading to warning of dead code.

What made the transition is easy was using the reqwest crate, and heavy use of serde_json’s json! macro. Comparing the Python and Rust version of the tests, makes them look very comparable.

Rust Version src:

#[test]
fn test_authenticated_user_cannot_modify_site_using_bad_request() {
    test_site();
    let api_base_uri = common::api_base_url();
    let client = Client::new();
    let auth_token = auth_token_headers();
    let response = client
        .put(api_base_uri.join("/api/site").unwrap())
        .body("")
        .header(Authorization(auth_token))
        .send()
        .unwrap();

    assert_bad_request_response(response);
}

fn assert_bad_request_response(mut response: Response) {
    let expected_response_json = json!({
        "error": {
            "status_code": 400,
            "message": "Bad request",
        }
    });

    assert_eq!(response.status(), StatusCode::BadRequest);
    let actual_json_response = response.json::<Value>().unwrap();
    assert_eq!(actual_json_response, expected_response_json);
}

Python Version src:

def test_authenticated_user_cannot_modify_site_using_bad_request(
        auth_token_headers, api_base_uri, test_site):
    response = requests.put(
        url=f'{api_base_uri}/api/site',
        data='',
        headers=auth_token_headers,
    )
    assert_bad_request_response(response)

def assert_bad_request_response(response):
    expected_response_json = {
        'error': {
            'status_code': http.HTTPStatus.BAD_REQUEST.value,
            'message': mock.ANY,
        }
    }

    assert response.json() == expected_response_json
    assert response.status_code == http.HTTPStatus.BAD_REQUEST

I am pretty happy with how everything turned out overall. My next bit of Rookeries work will be migrating away from invoke and make to cargo make.

Rookeries v0.10.0 – Rust Re-write

Posted on by

I just rewrote Rookeries in Rust, and the latest version is now available as a Docker image on Docker Hub. (This is why I have not responded to emails in a bit… I’ve been doing a lot of thinking of what I want to do next.)

So why a rewrite? Ultimately I decided to change the direction of Rookeries as a project (making it more of a static site generation tool + headless CMS). I also wanted to improve my knowledge of Rust. (A programming language that I believe is quietly revolutionizing computing bit by bit. I really need to write about it sometime) But to show that there is reason for my madness:

Traditional CMS

I realized that my approach to Rookeries was heavily inspired by WordPress. And WordPress represents the best in class (in terms of ease-of-use) for traditional CMS. Traditional CMS being applications that control all the aspects of a website: pulling data that it organizes inside a database, the applying logic to manage the workflows of the data, and the creating and managing the user interface that ultimately shows the data. The traditional CMS has to do a lot of things. That usually means that the CMS developer needs to add a lot of assumptions and constraints to ship said CMS. And when those assumptions no longer hold, it takes a lot to change the codebase.

In the case of WordPress (and similar styled CMSs), the server has to do a lot of work with building a page. A WordPress site consists of both the site, and its admin console. While there has been work on updating the theme management and the page/post editor, ultimately there are limits to what kind of UIs that WordPress can support. WordPress also has a massive plugin marketplace, because the core developers do not know what every website will need. (Nor does it make sense to build everything in or every possibility.) For a modern WordPress powered site to stand-out, it needs needs a custom theme and often a half-dozen plugins. Naturally having so many moving parts, from different vendors (some theme and plugin devs are better than others) means there is a a greater chance of security issues. The routine upgrade of themes, plugins and the WordPress core is a rite that one has to diligently perform regularly.

It would be impossible for Rookeries to compete with WordPress head on, just given the number of hours I would need to sink to make things a reality. Also I do not think this desirable given the current direction.

The Alternative of Static Site Generation and API Driven Sites

The assumption that the server must handle most of the layout and display issues does not hold when dealing with modern browsers. Modern browsers can run rich multimedia experiences and real-time applications. Running intensive applications such as a 3D game is now possible, and will become come place. Ultimately the web is now a platform, a target that application developers can directly target.

As a result, many web agencies are turning to the JAMstack and having their websites act as their own full-blown applications. The applications either talk to a headless CMS acting an API or even foregoing that with static site generation. Probably the best example of this is Gatsby for generating sites built in React, which I am currently using to build out the Amber Penguin Software site. I also plan on eventually converting my other existing WordPress sites to use Gatsby or a similar tool. These tools as other web agencies found, separate out the complex logic of a frontend app from the backend and its data. These means more flexibility when building out a site. It also means that the backend CMS can be hardened and simplified to avoid many security issues. In fact with static site generation, the web app can be so removed from its CMS and database, that attacking a site becomes irrelevant. Such a site can also have better performance, and take advantage of caching and CDNs.

The Future of Rookeries

While there are a number of headless CMS and static site generators out there, I feel like the overall experience of working with them needs some improvement. There are no real good headless CMS systems written in Python or Rust, and I do think there are some really unpleasant rough edges around running a static site. Hence I plan on reworking Rookeries to help me build a CMS that can power my various sites. At the moment, this version 0.10.0 has feature parity with the previous version of Rookeries. I still need to rewrite some of the admin and testing in Rust, to avoid the need for Python. Also I need to figure out a nice manner to expose a flexible API to my sites. Fortunately I have enough examples to keep me busy for a bit.

Rust, a New Toy Opening New Possibilities

Posted on by

I have not had a chance to blog in a while. Aside from the usual busyness of life, and the occasional bouts of illness, I have been distracted by a few new thing I’ve been learning: Rust, Kivy and Electron. I’ll write about Kivy and Electron in a future post. And a lot of that is centred around the upcoming product launches for Amber Penguin Software. But that is again for another post.

Rust

For the longest time, systems level programming (especially operating systems) have fascinated me. As part of that, I tried to learn the languages used to implement systems namely C and C++. While today I feel more comfortable with these languages, these still scare me with either their complexity (C++), their programming tools (gdb, gcc, autoconf, and minions) and their potential to do horrible things to your system if you are not careful. And bugs can be incredibly difficult to trace down and debug. So I while I have tried to code more C and C++, I still avoid them for these reasons.

Recently I started playing around with Rust, the new language developed partially by Mozillians. After trying to write a small Docker utility in Rust, and working through some koans for Rust, I have become smitten with the language. While some of the borrowing, references and types drive me a bit crazy, overall Rust is an amazing language. It brings the best ideas from Haskell, ML, Python and C and makes them very accessible. While the language is new and evolving the community has super active and creating lots of libraries for the language on crate.io.

Also I recommend listening to the New Rustacean podcast to learn Rust as well. It is not only informative, but very well executed by host Chris Krycho. So far I’ve listened to 10 episodes, and between the podcast, the koans and simply playing with Rust, I’ve learned a lot about Rust. In fact I feel more comfortable with Rust now then I have ever felt with C or C++.

Qt Bindings for Rust or Rather the Lack Of

On the topic C++, the one thing that keeps bringing me back to liar of C++ is the amazing platform that is Qt. However one on the most infuriating thing is that (aside from the limited QML), the only effective way to program Qt apps is using C++. Sure there is PySide and PyQt, with promises that PySide will actually support modern Qt5… some day… soon…

One of the things I hoped for Rust bindings for Qt to come quickly. Unfortunately it turns out that binding to Qt and C++ inside of Rust is really, really hard. There is an interesting project to develop some Qt C bindings for Rust, but it is really early in development and I had no luck using it. It probably is easier to bring in Rust to a C++ Qt project than the other way around.

In general UIs in Rust is a weak point for the language. Then again UI libraries are not the simplest thing to get off the ground, and it might be easier to rethink how we build them in general. Again this something I can get to in a future post.