PyCon US 2016 Talk – Pythons in a Container

At the end of May, I presented a talk at PyCon 2016, on using Docker with Python microservices. You can imagine the rush I felt getting to present on such a popular topic at such a large and important conference as PyCon! While it took me a while to recuperate after PyCon and Portland both of which were amazing. I would definitely do another talk at PyCon given the opportunity. Anyways I hope you enjoy watching the video of the talk! Below the video I also wrote about preparing for the talk, its reception, and a bit of the controversy that it stirred up below the fold. ūüôā (And I apologize for the lateness of this post, its been sitting in my backlog waiting to get finished for a few weeks now. ūüôĀ )

Video

Links

Abstract

Microservices and Docker are all the rage for developing scalable systems. But what challenges will you face when developing and deploying Python apps using Docker to production? This talk goes into the real-life lessons learned from creating, deploying and scaling Dockerized Python applications.

About the talk

Preparation for the Talk

PyCon talks definitely take quite a bit of time and effort to prepare. In my case, the talk took 3 major revisions before becoming the talk that I actually presented at PyCon. What started off as a intro to some of the concepts of Docker with some minor Python points, became more of a lessons learned targetted at intermediate to advanced developers. One of the things I wished I had (and I planned to but didn’t pull of) was to mention and thank my team for helping me preparing my talk. So thank you Kevin Qiu, Biniam Bekele, Yele Bonilla, and Gavin D’Mello for all your support, sitting me through three versions of my talk, and all the amazing feedback! I’ll make sure to include a slide with thanks next time.

Also I am very thankful for Jared Kerim from Mozilla, who presented at the local Python Toronto meetup about his team’s Docker setup. He was kind enough to let me use his example docker-django-template project. His example was also inspirational for my own docker-compose example using my CMS Rookeries. An example that I crafted and tested on the plane trip over to Portland. (After talking with other speakers, finishing up your presentation, notes and examples on the plane trip over is a proud traditional of PyCon and other conferences. :D)

Reception

Overall the reception of the talk was amazing! The talk turned out quite a crowd, in fact filling up most of the room. (I’m not sure of the capacity of the room but I estimate over 300 people attended). I was pretty nervous, but with the exception of a few stumbles, I think I pulled off the talk quite well. I really enjoyed some of the questions that were fielded during the Q&A session, and also privately afterwards. I wish could of answered some of the Docker Machine and Amazon ECS questions better, but I simply have not worked with both technologies long enough to give proper advice.

Controversy

The most surprising aspect of the talk was the controversy it stirred up. At the end of the Q&A you can hear some comments from a young lady about where I supposedly went horrbily wrong, and how there were tweets flying back and forth about it. I had turned off the notifications on my phone when I got up on stage, to avoid getting distracted. She persisted with telling (or trying to explain) what was wrong in the private gathering afterwards. Unfortunately she did not do a wonderful job of communicating, and I felt it took away time from others to ask their questions. It didn’t help her case that she admitted to being a novice at Docker. Please don’t that as an attendee, there are better ways to disagree and communicate that.

I later approached by a gentleman (thank you whoever you are), who mentioned I should go talk to the OpenShift guys since they had some concerns about my talk. News of the Twitter controversy worried me, because I hated the notion that I had gotten on stage and toled people to go and do the wrong thing. Especially when apparently I’m telling the opposite of what Glyph from Twisted said to do. After a brief chat (and a nice demo about their cool Kubernetes suite) from the OpenShift guys, I found out that Graham Dumpleton, the creator of mod_wsgi and who works on OpenShift had done a live tweeting commentary during my talk, where he disagreed with a few of my points. Long story short, eventually I was able to chat with Graham. He was a great sport and explained his points. Interestingly enough I had also talked with the folks at Docker. And they agreed with the points in my talk, and the logic behind my points. Essentially most of my points were based off the best practises they proposed.

Anyways I listed a few of Graham’s points with links to his blog posts (thanks again Graham!), and some of my quick thoughts on each one. A quick disclaimer about some of my points: the advice I gave worked for us in our datacentre, and that it might not work for others in other environments. It should work well, it might not be perfect, but it worked for us, and some of the folks at Mozilla. I gave a disclaimer at my other talk on a Ansible setup for WSGI apps at PyCon Canada, and I thought it was superfluous. But it turns out it is a useful thing to mention, and be explicit.

Errata

So the slide that caused a good portion of the controversy was the base image one. There I had provided an example Dockerfile on half the slide and discussed about base images and good Dockerfile practises, with points on the lower half. Now the example was meant as a toy and not necessarily complete. It is difficult, even impossible to present a well formated, perfect Dockerfile in that context. There is only so much room on a slide to fit both an illustrative example and some explanatory points. That is why I included links to some samples, that hopefuly did a better job of it.

Virtualenvs

Ah yes, the “enfant terrible” of my talk. ūüôā If you want to be controversial in your talk, mentioning something like this will get people’s attention. (Ironically, it was not my desire to stir up a controversy). Graham post a while back why you might want to use virtualenvs in your Dockerized app. It is a longish post, so I’ll give a shortened version. Basically when you base your image off some distro (say Ubuntu, Fedora or what not), there is a good chance of bringing in more Python packages in your system site packages than you expected. e.g. You’re building a Flask app, and the package maintainer included a version of Werkzeug in the base Python install, so now when you pip install Flask as part of your requirements you get the wrong version of Werkzeug.

And that is a valid point (with my example)… except if you use something like the official Python 2.7 base image… which installs just Python. I would argue that you would catch and resolve this issue, if you are auditing your Docker images. (And you should be always doing your due diligence and checking your base and resulting images. ) So yes… you don’t really need virtualenvs, but you can also use them if you are concerned that you might be getting conflicting packages.

Volume maps

Graham was right about the adding volume mapping in the Dockerfile being problematic. You should not define volume mounts in your Dockerfile, since they create extra files with sudo-like permissions on the host (see /var). In your own datacentre that isn’t a problem. A multi-tenant cloud provider like OpenShift, would disallow you to create those files. The documentation argument I provided is not all that useful, since you can document the mountpoints in the README that you would provide with the Docker image.

Base Images

Base images are hard to get right. And there is a lot of debate whether or not to use tooling instead of base images. Graham says his warpdrive tool will do that sort of a thing. At work we build out our own tooling for building “standard” service Dockerfiles, and that just add another level of abstraction. I prefer base images since it while not ideal, provides less levels of abstractions that can get in the way when you’re debugging your Dockerfile setup. But your mileage may vary here.

So yes, good base images are hard. Try not to build your own unless you find it really useful and you have a great base to work from.

Installing GCC/Build Tools

In an ideal world one ought not have to include GCC, Python dev headers and so on. Yes, one can pip install using wheels, but that doesn’t always work out.

Dockerfiles

Formatting of the RUN command. This is not one of Graham’s points, but it did come up. Yes, you should format the RUN commands, with a line for each command and using a \ line continuation for readability. My slide didn’t have enough physical space to do so. My Rookeries example does a better job of this.

Running as Root

Graham is right, you should not run containerized apps as root. That is a bad security practise that can lead to an attacker compromising your Docker host via a privileged account on your Docker container. Again a bad example on my part, I should of added a USER command and dropped the VOLUME line, or maybe rethought the use of an example.

UWSGI and the HTTP flag

No, you don’t need it and you should use the UWSGI protocol if you put an NGINX container before your WSGI container. I left the flag in to make sure the example Dockerfile was runnable. My bad on trying to get a good illustrative example, but it wouldn’t be a good idea in production unless you feel comfortable exposing UWSGI to the direct HTTP traffic.

Personally I’m not a fan of mod_wsgi + Apache, but Graham did point out he created mod_wsgi-express to simplify your life. If we continue to use Apache + mod_wsgi at work, then I’ll try to get us to use mod_wsgi-express too.

Final Thoughts

Anyways, I hope got everything right. Thank you for reading all the way to the end! ūüôā

Diving into the Deep End with Python

Where I Stand with Java

Most of my professional experience in software development has centered around Java. ¬†That should be no surprise as I learnt Java in university, and all the trappings that go with it: Javadoc, the Java Standard API, Eclipse and JUnit. ¬†My professional work exposed me to JSP, Servlets, SpringMVC, Struts, JDBC, TestNG, Ant and the Android SDK. ¬†Sure I’ve used other languages such as PHP, Perl, Scheme (mmm… functional programming…), C and Prolog. ¬†The mainstay of my experience is still Java.

All that said, even as I include more and more exotic features of Java as language (i.e. enumeration, annotations, reflection, etc.) and as a platform, I find myself more and more constrained by the language itself. ¬†Sure I feel comfortable with it, and I’ve written code personally that spans tens of thousands of lines. ¬†However I find a lot of the code is boilerplate code. ¬†Heck thanks to JAXB, I can quickly describe a series of classes using a XML schema. ¬†That is right, a XML schema, one of those awfully obscure XML trappings that only enterprise application developers really should ever care about (or those working with complex document-style data). ¬†Also a large Java codebase is nearly unapproachable with anything less than a powerful IDE. ¬†Thank goodness for Eclipse! ¬†(I write this wearing an Eclipse branded fleece jacket.)

Java is a wonderful and powerful language with great tooling.  However it just seems so wrong to have so much complexity.  I fear that this has more to do with the design decisions in the language itself than anything else.

Enter the Competitor: Python

I also learnt Python in university but I ended up using it largely for building system administration scripts rather than a general programming tool.  Instead of fighting with unwieldly BASH or Batch scripts for admin work, I use Python.  And I have created some powerful scripts using Python for tools for hunting down empty directories and rogue files for automated renaming.  After fighting with Java to do my bidding, I felt the need to look for some better language when building my new hobby project.  Also my recent attempts at learning Qt and C++ at the same time, have made me want to first learn the platform and then the language.  This is where Python seemed to hit a sweet spot.

While talking with one of my old university friends, Dan D’Alimonte, the topic of learning Qt came up. ¬†Instead of using C++ for his Qt apps, Dan used PyQt. ¬†After looking at his code, I decided to experiment with PySide and lo and behold I made a Qt app myself! ¬†On the web side of things I heard a lot of good things about the Django web framework. ¬†Hence I started experimenting with and investigating Python, PySide and Django. ¬†In addition to the simple can-I-do-this-with-that experiments, and playing around with the tutorials, I researched the tooling and performance of Python based applications.

I am impressed so far with what I have found out about Python. ¬†Coding in Python is a breeze and enjoyable. ¬†I can comfortably use any text editor to program in Python. ¬†I found that PyDev makes Python development while using Eclipse easy. ¬†Getting new packages for Python is simple, and I do not have to rely on a package maintainer. ¬†The standard libraries in Python are on par with Java’s. ¬†You can always extend Python with a C or Java binding. ¬†Building a desktop app using PySide turned out incredibly easy, compared to the standard C++ way of doing things. ¬†Django is proving to be a very capable, and apparently *gasp* fast web framework especially when run on the optimized PyPy version of Python. ¬†There is even mention of Python being able to build web frontends, at least in the documentation. ¬†And importantly for me, Python has various XML, database layer, testing, and other enterprise and mobile capable libraries that I can potentially use in my day-to-day work.

Now naturally the quirks of a language become more readily visible once you actually start working on a real-life project. ¬†That is my next step. ¬†Then I can figure out if Python could really replace Java as my go-to language. ¬†I will keep you posted once I get further in the project I am working on. ¬†In the meantime, I’ve collected a bunch of links related to my investigation with Python. ¬†Enjoy!

Python Related Links

Hello Again Identi.ca!

It seems that all the cool kids are using Twitter nowadays. ¬†Then again what passes as cool now sounds more like either being different for different sakes. ¬†Or that you use or do things that are “cool” or trendy or interesting to the mass public at the moment. ¬†Rarely the cool, trendy or different thing is actually interesting or innovative or world-changing as much as it something that is well marketed.

As for myself I’m on Twitter because there are a lot of people that are not on my favourite microblogging platform: Identi.ca! ¬†With Twitter being a bit more accessible on my Android smartphone and the smaller crowd on Identi.ca, I sort of stopped using it for a bit. ¬†However that will change. ¬†I prefer Identi.ca because it is libre software and my data is not held hostage in some corporation’s database. ¬†I’ve updated my Identi.ca handle, wired up my microblogging widgets back to Identi.ca (and Twitter). ¬†Now I plan on using it more often, and pushing my blog updates through Identi.ca first and then on to the other three popular social media that I use. ¬†If everything goes as planned y’all should be able to read this post in a few moments after I publish it… right now. ūüôā

Innovation in Increments

I have the good fortune of working in a Research & Development group. That means I get to learn about new ideas, experiment with them and apply them appropriately. Unfortunately I can not discuss my current project, other than it involves automating the creation of mobile applications. While I can not say that I am building something that is so deeply innovative that it has no precedence. But then again what most people do not realize that innovation happens mostly in small increments. You take an interesting idea, see if it makes your life easier and better. If not you review your work and options, and you try again. If it works, you get innovation!

So while I can not comment on my own work… ūüôĀ I can point some interesting work happening in the libre software community.

Canonical’s New Take on Scrollbars

Many of today’s computing innovations like tablets deal not with radical new technologies. ¬†But rather making technology more usable for non-developers and non-engineers. ¬†It might not sound like much, but Canonical is working on improving the usability of scrollbars in their Gnome desktop:¬†http://www.markshuttleworth.com/archives/615

Take a look at the video in Mark Shuttleworth’s post. ¬†I definitely think that abstracting the line indicator and the actual control is a great idea. ¬†It also makes it more touch friendly and intuitive.

MeeGo and Qt Lives for KDE and the N900

While not so much an innovation per se, I am happy to hear that the development of Qt and MeeGo will continue. ¬†The KDE crew came out and pointed out that Qt back when KDE started was a great framework and is even better now. ¬†Back when I started using KDE, I was amazed at how well everything integrated together in look and feel terms. ¬†This was all possible with KDE settling on one good UI framework, Qt. ¬†Now that it is more cross-platform and rounded out, it still is a great compelling framework to learn and use. ¬†There are some governance issues that need to get worked out, but it is nothing that won’t be resolved nicely soon. ¬†I indeed intend on learning Qt, as soon as my own schedule clears up.

[Another analysis on the Nokia/Qt/MeeGo/KDE question. ¬†Man isn’t life in the libre software world messy at times.]

As for MeeGo, sounds like Nokia will be supporting the N900 as an official development device for MeeGo. ¬†So maybe Mr. Elop changed direction, but at least there is a way forward for MeeGo handset developers. ¬†Hopefully that’ll mean that we can get started hacking on MeeGo. ¬†And once more devices come out, all developer efforts can get carried over. ¬†Maybe, just maybe we’ll finally have a good libre software platform for new¬†disruptive¬†devices, that won’t be threatened by the domination of one massive vendor. ¬†I’m looking at you Google, Microsoft and Apple.

Ontario GNU Linux Fest 2009

Tomorrow I’ll be going to Ontario GNU Linux Fest 2009.¬† It looks like this year it will be another year for this convention.¬† Not sure why the insistence of adding the “GNU” to the name…¬† Still I’ll be there, checking out as many of the presentations as humanly possible.¬† And doing a thorough raid of possible swag, even if it means opening up my wallet to cover “extra” costs. ūüėȬ† Anyways if you are in the Toronto area and a Linux fan, you should definitely check this con out.¬† See you there.


Ontario GNU Linux Fest 2009. Register Now!

New Site for justCheckers

I’m setting up a new site for the justCheckers project.¬† The only real good reason for my working at all on this project, is simply to build a portfolio.¬† As a software developer, I want to show to others that I can code.¬† And that my code is clean, elegant and professional.¬† I also want to show that I can write my own web applications.¬† So I am writing the justCheckers website as a PHP, MySQL DB enabled application.¬† All using proper XHTML, CSS and a touch of AJAX.¬† The site is a work in progress.¬† I’m not planning to build an entire flexible CMS, rather I plan on building a custom site.

The Story of the GNU

Last Thursday I received a package from the Free Software Foundation (FSF).  Along with a nifty t-shirt, I ordered my stuffed gnu.  Now the penguin on my desk has a new friend. The reason for the penguin is obvious, I am big fan of the GNU/Linux operating system. The penguin is the de facto logo and mascot of Linux.  But what is up with the gnu?

Well before Linux got off the ground or even existed, Richard Stallman (RMS) started the GNU project.¬† GNU stands for GNU is Not UNIX. Yes, GNU is a recursive acronym, one of the many cute jokes circulating in the hacker community.¬† The goal of the GNU project was to build a working and totally free (as in freedom) operating system.¬† RMS settled upon porting UNIX, not because UNIX was the be-all-end-all of operating systems.¬† Rather older versions of UNIX came with source code, and so that the new OS could be based off studying the way the old System V UNIXes¬† worked.¬† The project progressed well with the development of an entire toolkit: source editor (EMACS), compiler (gcc), linker (ld), and all the other necessary tools to build an OS.¬† Then came the difficult part of writing a kernel.¬† Unfortunately the original kernel (GNU Hurd) never got off the ground.¬† In fact to this day, the Hurd kernel is more or less in delayed development.¬† Fortunately at the time a kid in Finland-Linus Torvalds-started hacking on a kernel based off the Andrew Tannebaum’s MINIX source code.¬† Torvalds decided that the GPL would be an excellent license for his kernel.¬† And thus the dream of a fully viable free operating system started.

We have come a long way since those humble beginnings.  GNU/Linux looks like it will be the dominant OS of this century.  Also the ideals of free software are now fully realizable.  We still have a long way to go to running free software conveniently and comfortably, but we are getting there.  We should thank RMS and all the GNU contributers for building such a great free software toolkit.  Also thank Linus and the other kernel hackers for creating such a robust and flexible kernel.  Finally give a big thank you to all the contributers of the free software and open source movements for making this dream of free computing a reality.

justCheckers on Hold ‚Äď Again ‚Äď Kinda

justCheckers logo

justCheckers logo

I’m putting my involvement with the justCheckers project on hold for a while.¬† Progress ground to a halt when I started to dig into the code.¬† In its current state most of the application’s core functionality needs reworking.¬† Meaning to go forward someone would need to reimplement slides and jumps that allows for multiple jumps and so-called “flying kings”.¬† And the GUI needs refactoring to run in a multithreaded manner and with a main game event loop.¬† I already devised the algorithms for the core game engine.¬† But I need to translate that into real code.¬† I plan on implement those eventually.¬† But the amount of effort to reward doesn’t add up at the moment for me.¬† So justCheckers will not be on my high priority list for the next little while.

Just to be clear, I am not abandoning the project.¬† I still want to work on it.¬† But there are higher priorities on my list.¬† If anyone wants to step up to the plate and massage the code, I’ll gladly help.¬† And when once I get my other higher priority tasks done, I will return to hacking on justCheckers.

RMS Does Toronto

Richard Stallman, President and Founder of the FSF

Richard Stallman, President and Founder of the FSF

Yesterday I got to finally meet Richard Stallman (RMS) in person.¬† And yes, he is a way cooler dude than many in the “open source” gang say he is.¬† The FSF announced a while back in a press release, that RMS would be speaking at UofT on the topic of copyrights in a networked world.

RMS lectured on the history, current politico-business problems with copyrights and some measures to fix them.¬† I personally found his points interesting, and I’m interested in trying out some of his ideas.¬† The brief period of exclusive commercialization and modification authorship rights versus long term exclusive publication rights for non-technical documentation is also pretty good.¬† Also he briefly touched upon the need for a micro-donation payment method.¬† Overall, a very interesting and engaging talk to a large student audience.

At the end of his presentation, he auctioned off a stuffed GNU and “Happy Hacking” t-shirt.¬† Eventually the proposed prices became too steep for even myself, so I this morning I went on down over to the GNU/FSF’s online store and bought both for myself.¬† Getting back to the evening, I also got RMS to autograph my copy of “Free Software, Free Society“.¬† I highly recommend reading that book.¬† Also RMS asked me personally to stop using the terms: open source and closed source.¬† Not only are those terms misleading but they totally ignore the important issue of user and developer freedoms.¬† So henceforth I’ll try to use the terms “libre software” or “free software” instead of “open source”.¬† And “proprietary” for “closed source”.¬† Also I got meet to Dave, one of the organizers of the event and DrProject developer.¬† And Aaron one of attendees.

A big thanks to RMS, the FSF and the students who organized this event.

Open Source Gamer Goes to Ontario Linux Fest 2008

Last, last weekend (October 25) I attended my first conference, Ontario Linux Fest 2008. And what an awesome event it was! The event turned out be a somewhat low key event. Not a huge number of exhibitor booths, but the Eclipse, Fedora, FOSSology, FSF, OpenStreetMap and Drupal people had setup shop there. The event featured 4 tracks, and a number of great speakers. Got to meet Jorge Castro from Canonical, Ross Turk and Daniel Hinojosa from SourceForge, Jon “maddog” Hall, Jeremy Allison from Samba, Ian Darwin and Bradley Kuhn from the Software Freedom Law Centre (SFLC). It is always an amazing thing to meet the people “in the flesh”, who make a difference in the FOSS community, and read from blogs. The seminars were fantastic and inspire me to further my own open source/free software involvement.

One of the organizers, Richard Weait tried to convince me to start up a Brampton LUG. I might try it out, once I have a little bit more time. Nothing fancy mind you, just one night a month at the Coffee Culture, a few ads and a mailing list. We will see how it will go.

Also I got to go out to dinner with Bradley, the Red Hat and FoSSology folks. And thanks to Google for providing yet another free service in life, in the form of free drinks at the reception/ after-party.

So thanks guys for the awesome time. Sorry for not blogging earlier. Life keeps me busy as always. And lets make the 2009 event even larger!