Dear readers,
I was hoping to have a new entry for you this week. Unfortunately I am swamped with non-blogging work at the moment, and I need to concentrate on this for the next couple of weeks. So I’m taking a break from blogging for a bit. I should be back to my regular blogging schedule in the next couple of weeks.
Cheers,
Dorian
Apologies for missing last week’s scheduled post and being late with this week’s post as well. I’ve been putting off writing articles and refilling my queue, with other things that have been (or seemed) more important than blogging. Either way, I’ll try to fix this so that next week I’ll be back to my regular schedule. –Dorian
These past weeks right before the start of the new year, I have been experimenting with something new. As part of trying to use server-side rendering for React client inside Rookeries, I decided to figure out how to achieve this using NodeJS. To kill a couple of birds with the same stone, I decided to use this as an opportunity to play around with ES6. In these next few blog post I will write about some of the lessons I learned along the way.
To help me focus my learning, I decided to concentrate around a project that would provide a skeleton for my learning. I choose to recreate one of my earlier Flask project, which runs the Amber Penguin Software website. This web application acts as a cross between a static file website and a CMS, by serving template pages that render Markdown into the body of each page. The routing is a fairly trivial look up of flat files, and returning a 404 error page when a page is not found. The tech stack being Flask, a simple [J]inja2](http://jinja.pocoo.org/) template and Markdown as the Markdown rendering engine.
My project consisted of four phases:
The first task consisted of figuring out what NodeJS technologies I could use to recreate the Python/Flask app. Turns out that the language specific communities in the web app world like to borrow heavily from each other. Just as Ruby’s Sinatra microframework inspired Python’s Flask, so did Node’s ExpressJS take notes from Flask. Jinja2 inspired Mozilla’s Nunjucks and a bunch of other similar templating libraries. (I ended up using Nunjucks since it is the most mature library) Marked replaced Markup. The tricky part was actually replacing Python’s io.open() to open files. With a bit of experimentation I figured out how to use Node’s fs (file system) module and its readFile() and readFileSync() methods.
In short I could translate the tech stack this way:
Next time, I’ll go into the details of setting up the ExpressJS apps and routes.
For the past couple of days instead of working on actual development work related to any of my projects, I’ve been transferring all of my domains from DreamHost, my old hosting provider to a new DNS provider. I was looking forward to a gentle switch over, to my new Canadian (eh!) DNS provider easyDNS. Unfortunately like many technical problems, I ended up spending more time and effort than I expected to originally. (Enough effort that I’m late with posting this blog update today.)
It turned out that DreamHost made enabling my desired setup real easy, and hid a lot of the technical difficulties of setting up DNS records. easyDNS is a lot more flexible, but then I’m not a DNS record expert so getting a similar setup was tricky. Fortunately the fine folks at easyDNS are really responsive by email, and a few emails back and forth we arrived at a setup that worked nicely. Most of this came down to not understanding the terminology and not checking the right places
This the main record that maps a domain name to an actual IP (v4 and v6) address. In my case this would be the IP address of the server hosting all my webapps.
my_domain.com ---> 123.456.789.10 (not a real IP that I own)
Canonical Name or Alias, this is used to map a subdomain (e.g. www, app, etc.) to another part of the domain or another top-level address. This retains the subdomain name in the address bar of your browser.
www.my_domain.com ---> mydomain,com
mail.my_domain.com ---> my_email_provider.net
Redirects (usually via a HTTP 301 Redirect) an address to another domain or location. (This turned out to be the option I needed for most of my sites) A redirect naturally will change the URL in the browser’s address bar.
my_domain.ca ---> http://my_domain.com/
my_nifty_sideproject.net ---> http://side_project.github.io/landing/
Redirecting using an IFRAME, and retains the original address in the browser. Otherwise it works like a URL redirect, it just won’t look like one.
My original setup at DreamHost basically pointed all my additional domains at my main domain, and removed the www subdomain that normally gets tacked on to an URL by browsers.
dorianpula.com -> dorianpula.ca
www.dorianpula.ca -> dorianpula.ca
Now traditionally you are supposed to use a CNAME for the second example. I just ended up using a URL redirects everywhere to make things simple. And a A DNS record for the main top-level domain to point to my Linode servers.
The setup turned out real simple, but at first I could not get any of my changes to work. Or rather some of them works, other did not. It was very frustrating at first, but then the easyDNS support rep pointed out that I had not updated my nameservers for some of the domains names I transferred over. I was originally pointing to DreamHost’s rather than easyDNS’s nameservers and my changes simply were not propagating through. Once I fixed that, everything started updating as expected.
Finally I just want to make some recommendations for anyone looking for a hosting or domain provider. I started off using DreamHost, after migrating away from GoDaddy, and I was happy with them for the longest time. They are convenient, easy to setup especially for PHP apps and pretty supportive. I highly recommend them if you have a normal website (like a WordPress blog) and want to have one stop shop.
Personally I outgrew DreamHost, when I needed something more configurable for my Python webapps. I’ve since migrated to Linode, who provide very nice, configurable and affordable VPS (virtual private server) hosting. I love using them and they support a wide variety of different OS platforms and versions.
Finally I recommend easyDNS. Their great for Canadians, supportive and care about your Internet freedoms (their takedown policy is that you have to have a real court-order or being doing something blatantly illegal rather than some flimsy takedown letter from some random legal department). I really recommend them if you want a flexible DNS/domain hosting. The problems I encountered were my own doing and lack of understanding, that the support rep helped me resolve in a few hours and after few tries.
Well it looks like 2016 is off to a nice start. This weekend I submitted two proposals to talk at PyCon US 2016! And yes, I have already bought tickets, even though I am not quite sure how I’ll get to Portland, Oregon. 😛
But I thought why not? I had a lot of fun at PyCon last year in Montreal, and I’ve never been to Portland… and talking at PyCon Canada was actually quite fun. Now I don’t know if my proposals will get accepted or not. I will say that writing proposals is not fun. But fingers crossed maybe I’ll see you guys in Portland this spring… and maybe I’ll get a chance to talk as well.
…it makes me want to reflect on this past year and plans for the new year.
Professionally it has been a great year. I feel like an active and important part of the work we do at Points. Over the past year, I’ve gone for being mediocre in Javascript to someone who feels confident in working with that language. I have given talks both small and large, and people seek out my expertise. (Even though I don’t feel like I’m an expert at times in many things.) I have approached languages and frameworks like Erlang, Qt, C and OpenGL that in the past felt unapproachable, and have learned quite a bit about them. In general things are looking up, and hopefully next year I’ll continue this trend and expand my knowledge and experience in new directions.
Personally it has been a year of highs and lows. I am happy to feel that I’m on track with some of my personal goals. A large one was giving a talk at a PyCon, and that went pretty well. The steady progress on Rookeries has also been encouraging. I feel my faith has deepened which is an important aspect of my life. Also the realization and confirmation that I can be an interesting and fun person, that others want and seek out being with me, makes me feel better as a person. It has not been all been sunshine and roses, having to close down projects (like justCheckers) and give up on certain plans certainly has been disappointing. Also I have the nagging feeling of having not done enough writing. Hopefully I will rectify that, this upcoming year.
Maybe it is just a feeling; but seeing myself take on older projects and realize them feels empowering. There is still so much I want to do, but as this year draws to a close, I feel that I’ll be able to continue progressing forward. And I know this a bit premature, but since we are so close to the start of 2016: Happy New Year!
Today is Christmas Eve, and I just wanted to wish you dear reader: a very Merry and blessed Christmas to you and your family. And may the One who came into the world to save humanity, may also come into your life.
I recently upgraded my version of Ubuntu to 15.04. In the process, I found out that my init system had changed from Upstart to systemd. While having something as fundamental as the init system management was a bit annoying, it isn’t as bad as some folks are making it out on the Web. Here are some of the things I learned as worked with systemd in fixing my CouchDB server. Part of this is based on this excellent guide on using systemd for Upstart users. The other part is just experimentation on my part.
sudo systemctl status
Hint: feel to grep through the output to find anything
At first I could not find the CouchDB service. I had to uninstall the package and purge the packages and then reinstall it:
sudo aptitude purge couchdb
sudo aptitude install couchdb
After that the service showed up in the systemd services, rather than just having an Upstart service.
sudo systemctl status $MY_SERVICE
sudo systemctl [start|stop|restart] $MY_SERVICE
This was an interesting thing in systemd. Normally one has to look at the Upstart log at /var/log/upstart/$JOB.log. systemd provides its own logging mechanism, so to see the log of a service you have to use the journalctl utility.
sudo journalctl -u $MY_SERVICE
The output behaves just like one would expect from less.
Once I got over my initial head scratching of how to use systemd, it was not bad. Rather it feels different, but not in a bad way honestly. I might look into this more closely and see if I prefer using something like systemd over supervisord for controlling even WSGI apps.
The one disappointing thing that I discovered related to my experiments with systemd, or rather specifically with CouchDB in Ubuntu 15.04. There doesn’t seem to be a way to configure CouchDB to have admin users with passwords for some reason I can’t quite fathom. In the meantime, I guess I’ll have to stick to running CouchDB from a Docker container until I can resolve the issue natively.
We use SauceLabs at work to handle the maintenance of environments specifically for running web tests. I highly recommend using SauceLabs over trying to maintain your own internal testing lab. (I know I helped create our Android web test environments and maintain the iOS Selenium environment. It was a lot of work and not a lot of fun especially with deadlines breathing down my neck.)
As mentioned in a previous post, my started using Splinter for writing web tests. Splinter has excellent support for running remote Selenium tests, and hence support for running tests in SauceLab virtual VMs. However one thing I drove me nuts is that when looking at the SauceLabs dashboard, I had no idea if a test scenario succeeded or not.
It turns out that a SauceLabs VM has no notion of a successful or failed test… or more correctly a Selenium web driver doesn’t. But you can correlate a SauceLab VM test run (job) with a successful or failed test.
The only issue is that the example given uses nose and selenium rather than lettuce and splinter. So how did I get it working for splinter and lettuce? By adding the following code after a scenario is run:
@lettuce.after.each_scenario
def report_to_saucelabs(scenario):
#Remember to quit your webdriver after you run your tests!
scenario.browser.quit()
# This was the most sure fire way to check if this was a remote web driver vs a local one.
if not isinstance(scenario.browser, splinter.driver.webdriver.remote.WebDriver):
return
# Now gather all the failed steps and figure out what status needs to get sent
failed_steps = [step for step in scenario.steps if step.failed]
test_successful = bool(failed_steps) == False
# Finally we have to connect to SauceLabs using the saucelabs Python client (pip install sauceclient)
sc = sauceclient.SauceClient(SAUCELABS_ACCOUNT, SAUCELABS_ACCESS_KEY)
sc.jobs.update_job(scenario.browser.driver.session_id, passed=test_successful)
As you may know, I presented a talk at PyCon Canada this year. And as promised I wanted to post the video and the associated content here. Hope you enjoy it!
Intermediate level talk about migrating a Fabric deployment to a more flexible setup using using Ansible and Invoke. Follows the journey of changing the deployment of a Flask based blogging app from a Fabric script and pre-provisioned server to a modular system with Invoke tasks and provisioning using Ansible. Discusses the advantages and cons of moving to a declarative system versus direct shell commands. Touches upon on Ansible Roles, Ansible Galaxy and Invoke.
Testing the client side of Rookeries, has proven to be quite a challenge. Not necessarily because testing well-written React JS components is hard. Rather I found it hard to setup a proper and consistent unit test infrastructure to do so. Rather than going through the pain of writing and maintaining functional tests in Javascript, I decided to take a different path.
I wanted to write my tests in a business-domain-driven (BDD) style. While most developers find awkward to use at first, it is a great way to write out the business functionality and features of an app. It also forces you to think about your app in a non-technical manner, and prove to yourself (and others) that it does what you claim it does.
At work my team has been burned by slow and awkward web tests. Namely we worked with the Robot Framework, which uses its own DSL and is very difficult to work just as we wanted. Debugging tests was also quite unpleasant.http://lettuce.it/
Fortunately one of our newest team mates Kevin Qiu introduced us to lettuce (a BDD framework) and splinter (a Pythonic Selenium interface). And we’ve had a fair bit of success describing scenario using these tests. I won’t lie, Selenium is always temperamental. However the current batch of web tests have been very stable, and has very much convinced me that this setup can work effectively.
Rookeries usea a similar setup to what we’ve done at work: namely I use splinter to interface with Selenium. However unlike work, since Rookeries uses pytest instead of nose, I ended up using pytest-bdd to provide the BDD framework for Rookeries. Furthermore, I am using the pytest integration for splinter, to provide some of the browser fixtures needed for the tests.
Feel free to check out the functional tests in Rookeries to see examples of the tests. I also highly recommend watching Dylan Lacey ‘s talk about using Splinter for web testing at PyCon Australia 2013.