Rookeries v0.16.0 – Cross-Platform Support

On Wednesday night, I released version 0.16.0 of Rookeries, my developer/designer friendly static site generator! What makes this release exciting is that I added cross-platform support for macOS and FreeBSD! Also I setup an cross-platform installer, that works on all 64-bit x86 Linux distros including Windows Subsystem for Linux (WSL) 2.0. And I plan on supporting Linux, macOS (OS X) and FreeBSD as first-class systems going forward.

Also a multitude of stability and UX improvements landed in this release.
You can download Rookeries and install it using the new cross-platform installer here.

Rookeries v0.16.0 screenshot

Now this took quite some time and effort, especially around figuring out cross-compiling and what each OS could reasonably support. Over the course of the next few weeks, I will be writing about the challenges and how I overcame them when adding cross-platform support. (Including a quick aside why I dropped native non-WSL Windows support, and took up FreeBSD instead.)

Do you know that my mailing list is the first to know about releases and developments around Rookeries? Consider subscribing to the mailing list to be among the first to know!

Rookeries v0.15.0 Released – Technical Preview of the Static Site Generator

It turns out that rewriting a CMS into a static site generator takes a bit of time. 😉 After 3 months of silence, I can finally announce the long awaited release of version 0.15.0 of Rookeries. This release is the first fully functional technical preview of the static site generator!

You can now create, build and develop a full site with Rookeries! In fact I rebuilt the Rookeries site, using Rookeries to prove out the concept. You start by using the generator to setup a new project. Then you can add Markdown files to build new pages to an existing site, change the HTML and CSS templates, and run a build which creates a modern site. A site that in essence is a server-side rendered single-page application. So if you are running Debian or Ubuntu Linux you can try it out Rookeries by installing the deb package.

Now, this release is a technical preview release to demonstrate the concept. I plan on making the template and plugin systems more flexible. Also I want to improve the usability by adding an automatic rebuild when a page is edited. And I need to write proper documentation as well. Support for other operating systems and distributions is also in the works. However already Rookeries is significantly faster than its nearest competitor: GatsbyJS and smaller in size. And you get a nicer, more maintainable site than from many other tools.

So please take Rookeries for a spin, and let me know how you like it!

Rookeries v0.14.0 Released – Tech Preview of Plugins

I’m excited to announce today’s new release of Rookeries: version 0.14.0. This release previews support for plugins!

After quite a lot of experimentation, and I was able to create a tech preview of a plugin architecture. The server can now find and load frontend plugins. And the frontend plugins now extend the markdown editor, and provide new web components on a page. While the current plugins are essentially tech demonstrators, and the server and admin console need work, I am confident that this architecture will enable Rookeries to move to the next level.

What is the next level? I want to make Rookeries the most usable, extendable Markdown based CMS, targeted toward more technical users such as developers, and technical documentation writers. Rookeries will focus on enabling the creation of web site by writing Markdown, styling with CSS and using plugins for more complex additions: such as SEO, custom markup, image resizing, et cetera.

As for my next steps, I will be working in 3 areas:
* Reducing the size of the Rookeries Docker image (currently at a massive 4 GB).
* Refactoring the main app and the admin console to take advantage of plugins.
* Creating an experience for using plugins that rivals WordPress.

There is still a long way to go, however I am confident that Rookeries will be usable for production (outside of hosting its own site) before the end of the year.

Rookeries v0.13.1 – Release + Web Assembly Experiments

This past month I’ve been trying to determine the future direction I want to take my projects: especially Rookeries. Ultimately I want to build something that lasts and is able to sustain itself financially. As someone who is still a novice in the arts of marketing, and business development, this is hard, and doing this in an open, transparent environment like with libre software, just adds to it. While I have many musings on what is needed to run a sustainable open source project, they are just that: musings. Without getting into the weeds, I think the Ghost blogging platform team is doing the right thing. I recommend the Indie Hacker podcast that interviews John O’Nolan of Ghost.

Also I’ve been playing around with some of the interesting Rust Web Assembly projects out there. I think they have potential to improve Rookeries in a radical manner. However before I get anyone’s hopes up, I want to get something working before I announce the future direction of Rookeries. (Something, something, Web Assembly, something, something, plugin extensibility.) In the meantime I am releasing version 0.13.1 of Rookeries!

This version is mostly bug fixes, and code re-organization that will help me extend the project into the future.

P.S. My mailing list knows about my releases and plans of Rookeries, before the blog. So you should totally sign up for the Rookeries mailing list.

Rust + Gtk = Wow

As I’ve been experimenting with writing Rust apps, I attempted to create a small little GUI application. At first I attempted to setup everything with Qt, but C++/Qt interoperability with Rust is painful. Very, very painful. I experimented with some more radical UI frameworks such as Azul and Conrad. These have a lot of promise going forward. However for the here and now, I recommend looking at gtk-rs, Rust binding for Gtk.

With a bit of experimentation, looking through gtk-rs examples, some other projects using gtk-rs, and lots and lots of searching, I was able to create this:

This is a code viewer that lets you open Rust code, and view it. Yes, you are looking at a portion of the code that runs that code viewer. Getting the GtkSourceView working took some coaxing. Also I had to learn how to use the Glade UI editor for Gtk. Overall it took me about 2 to 3 hours to pull this off. I am very impressed with the results, and it opens up new possibilities for me.

Packaging up a Rust Binary for Linux

Prologue

I should of written an update for quite some time. While I’ve been experimenting with marketing analytics, learning about data science, business development, doing DevOps with GitLab CI and various other things, I wanted to write up my learning when I had a chance to internalize everything. However what made me decide to write an update is this tweet from Chris Krycho. Chris runs the amazing New Rustacean podcast, which is a must listen for anyone interested in learning about programming in Rust.

How does one package a Rust app?

Chris asked about finding a good way to distribute Rust binaries across Linux distros:

Interestingly enough, I recently figured out how to package a small Rust CLI utility that I’m working on. My response was:

This post elaborates on what I meant with my reply.

Building snap packages using snapcraft

The fine folks at Canonical (the makers of Ubuntu Linux), have created something called snap packages. These packages and associated package manager help developers distribute applications (desktop, cloud, etc.) in a safe, isolated manner. I currently have slack installed this way. snaps isolate apps by having the package maintainer declare the capabilities an app requires (network access, access to system files, the GPU, home directory, etc.), and then ensuring the apps can not escape this sandbox.

Basic Setup

Getting setup with packaging a Rust crate was not too hard:

  1. Install snapcraft: sudo snap install snapcraft --classic
  2. Create a snap template inside your crate project: snap init
  3. Edit the generated snap/snapcraft.yml as per the documentation.
  4. Build the snap using snapcraft.
  5. Install the resulting snap with sudo snap install my-cool-rust-bin_x.y.z.snap --devmode --dangerous (this assuming you are experimenting with building a snap)
  6. Add you should be able to distribute your app as a snap now. (See the caveats below.)

Caveats working with snaps

Now there a bunch of caveats when working with snaps. And for my own Rust utility, I found these too taxing and I decided to go with creating a standard Debian package instead. However if I wanted to target multiple distributions and my app didn’t have a very unorthodox setup (my app relies on using the Chrome WebDriver to control a networked device managed by dd-wrt), I would probably have gone with a snap instead:

  • You need to know what capabilities your app needs: file access, network, etc. and you need to declare the appropriate interfaces in your snapcraft.yml
  • Using something other than my local system (be it a Docker based build or using a different base like base18), failed terribly at least for Rust.
  • Whether or not I’d have more success if my base system was the recommended Ubuntu 16.04 and not 18.04 is an outstanding mystery.
  • The snap confinement, even on the much more liberal devmode, works very well. No amount of coaxing on my part, let me use system paths when trying to spawn a process. This could just be me though, as not declaring network access did not block my app.
  • The docs could of been clearer about what was the latest recommended approach. (Still way clearer than the documentation for creating a DEB or RPM from scratch.)
  • Knowing which libraries (for the type of base system) your app needs takes a bit of experimentation. (e.g. I needed libssl1.0 for some builds and libssl1.1)
  • I have no idea how the classical confinement should work, and it is not recommended either way.

The end result for me was a working snap package, but an app that would not work when called from an installed snap. However I think snap packages are probably the way to go moving forward (or a similar format like flatpak). Since I only plan on targeting local Ubuntu 18.04 installs, I ended up creating a Debian package instead.

Building a Debian package with Cargo

I found a nice utility for creating a deb out of a Rust crate, called cargo-deb. After installing the crate with cargo: cargo install cargo-deb, I simply ran cargo deb and I was done. cargo-deb looked into my Cargo.toml for the metadata, ran a build and a few moments later I was the proud owner of a Debian package. Since my app relies on the chromium-browser and chromium-chromedriver packages, I added a small section in my Cargo.toml as so:

[package.metadata.deb]
depends = "$auto, chromium-browser, chromium-chromedriver"

The $auto is something that the Debian packaging mechanism needs, and that is the comma separated format that DEBs use.

Building a RPM from a DEB package

Now this the part that I didn’t do this time around. However I figured out how to create RPM packages from DEB packages a few months ago. The trick is to use the alien utility to create a RPM out of a DEB:

sudo alien --to-rpm --scripts --verbose my-cool-rust-bin.deb

For the record, I did not try to improve or debug the resulting RPM. (This entire effort was for a product that failed to launch.) However as part of my tests I was able install it and run it from on CentOS VM.

Epilogue

Anyways, I would recommend the cargo-deb and alien approach, if you are not planning to distribute a Rust app across a multitude of Linux distros. I would recommend dipping into snap if you plan on distributing something more commercial and wide-spread like a slack or kubectl. And I hope that helps you on your Rust app packaging for Linux journey!

Embedded Rust Library Experiment for Python and Web Assembly

With my ever growing list of things that I need to catch up (like wiring my home network and managing Rookeries), I needed a small fun project that I can work on. Ever since I learned enough Rust to be able to convert Rookeries, I wanted to play around with being able to speed up my code with a Rust library. I am especially interested in figuring how to call Rust code from Python or from JS with Web Assembly.

As a test bed (and a reason) for me to learn this, I created a small little library for getting the uptime of a local server (Linux only): embedded-uptime converting between different measurement units like Celsius and Fahrenheit: embedded-unit-converter. If you’d like to follow along, feel free to check it out. I will be posting updates on the blog, and on the Rookeries mailing list.

Updated on 2019 February 4: When I setup the project, I forgot that server uptimes that rely on accessing a server’s /proc/uptime can not possibly work in Web Assembly in browser environment. After some consideration I decided to go with something simple that accessible from any platform, namely conversion between different units of measure.

Mailing List Migrations

Hello dear reader!

I hope you are having a wonderful year so far! You may have noticed a small change in the sidebar menu, where there used to be 2 sign-up forms. I removed those as there are dedicated pages to the signups now, and the sidebar was just getting quite busy. (I plan on doing a major redesign of the site this year. Stay tuned for details.)

Also you may have noticed that I have not posted a update recently. This is because I dedicated the end of last year and the beginning of the new year to organizing myself better. I hope to write more this year, maybe even on a regular schedule. While I might not be able to emulate Jonathan Stark’s one update a day for his mailing list, as I dear reader, unlike Jonathan I still have a regular full-time day job that I need to plan around. I do plan on updating my mailing lists for Rookeries and Juggling JSON with jq e-course once every 2 weeks. If I can spare the time, I want to return to releasing a blog entry on a more regular basis.

If you are wondering how I can manage all of this, I will say that Drip makes email list management painless. (It even managed to save me from some junk e-mail addresses which Gumroad did not…) Like with any sort of automation, it took me some time to set it up and understand how everything fit together. I hoped that more people would of migrated over from my Gumroad mailing list, but there I don’t know how much of those emails were junk. Most of them were, which gave me a lot of false hope about having an overwhelming crowd lining up to buy my e-course on jq. Drip gives me a much more realistic view. But I am still optimistic. Overall I am quite happy. What would make me even happier, dear reader, is if you signed up for one of my organic, free-range, and informative mailing lists! But no sweat!

Onward with Postgres and Diesel – Rookeries v0.13.0

That took longer than expected… but I’ve finally migrated Rookeries to use PostgreSQL 10! I finished the majority of the raw migration work about a week before PyCon Canada. However I did not want to confuse my updates and tweets related to PyCon Canada with my work on Rookeries. Also I wanted to add a few bits of polish, create a mailing list for Rookeries and fix up any issues I found with the site. (Quite a few data and TLS certificate issues happened along with this release.)

I found that migrating Rookeries from CouchDB to PostgreSQL quite challenging. This adds yet another data point to those that argue that fixing data models and databases can be significantly harder than fixing applications. One of the challenges I stumbled in was mapping the various keys in a CouchDB document for a resource, into a table. Fortunately the JSONB columns allowed for the flexibility to let me resolve some of my earlier problematic data designs later on. I also needed to familiarize myself with diesel.rs, the amazing ORM/database engine library for Rust. I struggled with understanding how custom types worked in Rust, and why they make database mapping of columns easy for more advanced mappings (like optionally nullable, JSONB and Array). However I managed to figure out even the hard parts using a combination of the documentation and by peeking into the test cases for diesel. Having a migration tool in the companion diesel-cli utility, provides the nicest UX for building out database migrations and mapping them to code. Overall the effort paid off and I am convinced that Rust has one of the cleanest and maintainable ORM/database libraries I’ve seen.

As mentioned earlier, I created a mailing list dedicated to Rookeries updates. I will continue to blog about releases, but if you are interested in the internals and development of Rookeries, then this mailing list a good thing to follow. Consider it like a journal that you can also reply back to and enter in a conversation about the development of Rookeries. I added a panel to the side of the site, that makes subscribing to Rookeries easy.

This will hopefully will be the last of the major technological shifts for the project, at least on the backend. The next releases I plan on concentrating on getting Rookeries in a state where I can drive Gatsby powered static sites via REST APIs. Once that becomes a reality, I will work on replacing my WordPress sites with Rookeries powered sites.

Fixing Docker on Linode (Linux v4.18.16)

This week as I had some downtime after PyCon Canada, I started working on resolving all the issues that I postponed. One of these issues involved applying security updates to my Linode server and rebooting the server. However when I did so… I noticed that the Rookeries site went down. When I logged into the server, I quickly found the problem: Docker refused to start after the kernel updates.

As this bug report on Docker for Linux says, there is an issue with the latest Linode kernel when it comes to OverlayFS.

This causes the containerdservice that docker-ce is dependent on to not start. When looking at the logs (using sudo journalctl -xe), you’ll see an error along the lines of:

modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/4.18.16-x86_64-linode118/modules.builtin.bin'
modprobe: FATAL: Module overlay not found in directory /lib/modules/4.18.16-x86_64-linode118)

Thankfully there is a workaround to resolve this problem. From the instructions you need to an override configuration for the containerd service:

$ mkdir -p /etc/systemd/system/containerd.service.d/
$ cat << EOF > /etc/systemd/system/containerd.service.d/override.conf [Service]
ExecStartPre=
EOF
$ systemctl reload
$ systemctl restart docker

Anyways, I hope this helps if you run into the same situation.