Category Archives: Web and Cloud Technologies

The Web and the cloud are the dominant features of the software world. This is their spot on my blog.

PyCon US 2016 Talk – Pythons in a Container

Posted on by

At the end of May, I presented a talk at PyCon 2016, on using Docker with Python microservices. You can imagine the rush I felt getting to present on such a popular topic at such a large and important conference as PyCon! While it took me a while to recuperate after PyCon and Portland both of which were amazing. I would definitely do another talk at PyCon given the opportunity. Anyways I hope you enjoy watching the video of the talk! Below the video I also wrote about preparing for the talk, its reception, and a bit of the controversy that it stirred up below the fold. 🙂 (And I apologize for the lateness of this post, its been sitting in my backlog waiting to get finished for a few weeks now. 🙁 )

Video

Links

Abstract

Microservices and Docker are all the rage for developing scalable systems. But what challenges will you face when developing and deploying Python apps using Docker to production? This talk goes into the real-life lessons learned from creating, deploying and scaling Dockerized Python applications.

About the talk

Preparation for the Talk

PyCon talks definitely take quite a bit of time and effort to prepare. In my case, the talk took 3 major revisions before becoming the talk that I actually presented at PyCon. What started off as a intro to some of the concepts of Docker with some minor Python points, became more of a lessons learned targetted at intermediate to advanced developers. One of the things I wished I had (and I planned to but didn’t pull of) was to mention and thank my team for helping me preparing my talk. So thank you Kevin Qiu, Biniam Bekele, Yele Bonilla, and Gavin D’Mello for all your support, sitting me through three versions of my talk, and all the amazing feedback! I’ll make sure to include a slide with thanks next time.

Also I am very thankful for Jared Kerim from Mozilla, who presented at the local Python Toronto meetup about his team’s Docker setup. He was kind enough to let me use his example docker-django-template project. His example was also inspirational for my own docker-compose example using my CMS Rookeries. An example that I crafted and tested on the plane trip over to Portland. (After talking with other speakers, finishing up your presentation, notes and examples on the plane trip over is a proud traditional of PyCon and other conferences. :D)

Reception

Overall the reception of the talk was amazing! The talk turned out quite a crowd, in fact filling up most of the room. (I’m not sure of the capacity of the room but I estimate over 300 people attended). I was pretty nervous, but with the exception of a few stumbles, I think I pulled off the talk quite well. I really enjoyed some of the questions that were fielded during the Q&A session, and also privately afterwards. I wish could of answered some of the Docker Machine and Amazon ECS questions better, but I simply have not worked with both technologies long enough to give proper advice.

Controversy

The most surprising aspect of the talk was the controversy it stirred up. At the end of the Q&A you can hear some comments from a young lady about where I supposedly went horrbily wrong, and how there were tweets flying back and forth about it. I had turned off the notifications on my phone when I got up on stage, to avoid getting distracted. She persisted with telling (or trying to explain) what was wrong in the private gathering afterwards. Unfortunately she did not do a wonderful job of communicating, and I felt it took away time from others to ask their questions. It didn’t help her case that she admitted to being a novice at Docker. Please don’t that as an attendee, there are better ways to disagree and communicate that.

I later approached by a gentleman (thank you whoever you are), who mentioned I should go talk to the OpenShift guys since they had some concerns about my talk. News of the Twitter controversy worried me, because I hated the notion that I had gotten on stage and toled people to go and do the wrong thing. Especially when apparently I’m telling the opposite of what Glyph from Twisted said to do. After a brief chat (and a nice demo about their cool Kubernetes suite) from the OpenShift guys, I found out that Graham Dumpleton, the creator of mod_wsgi and who works on OpenShift had done a live tweeting commentary during my talk, where he disagreed with a few of my points. Long story short, eventually I was able to chat with Graham. He was a great sport and explained his points. Interestingly enough I had also talked with the folks at Docker. And they agreed with the points in my talk, and the logic behind my points. Essentially most of my points were based off the best practises they proposed.

Anyways I listed a few of Graham’s points with links to his blog posts (thanks again Graham!), and some of my quick thoughts on each one. A quick disclaimer about some of my points: the advice I gave worked for us in our datacentre, and that it might not work for others in other environments. It should work well, it might not be perfect, but it worked for us, and some of the folks at Mozilla. I gave a disclaimer at my other talk on a Ansible setup for WSGI apps at PyCon Canada, and I thought it was superfluous. But it turns out it is a useful thing to mention, and be explicit.

Errata

So the slide that caused a good portion of the controversy was the base image one. There I had provided an example Dockerfile on half the slide and discussed about base images and good Dockerfile practises, with points on the lower half. Now the example was meant as a toy and not necessarily complete. It is difficult, even impossible to present a well formated, perfect Dockerfile in that context. There is only so much room on a slide to fit both an illustrative example and some explanatory points. That is why I included links to some samples, that hopefuly did a better job of it.

Virtualenvs

Ah yes, the “enfant terrible” of my talk. 🙂 If you want to be controversial in your talk, mentioning something like this will get people’s attention. (Ironically, it was not my desire to stir up a controversy). Graham post a while back why you might want to use virtualenvs in your Dockerized app. It is a longish post, so I’ll give a shortened version. Basically when you base your image off some distro (say Ubuntu, Fedora or what not), there is a good chance of bringing in more Python packages in your system site packages than you expected. e.g. You’re building a Flask app, and the package maintainer included a version of Werkzeug in the base Python install, so now when you pip install Flask as part of your requirements you get the wrong version of Werkzeug.

And that is a valid point (with my example)… except if you use something like the official Python 2.7 base image… which installs just Python. I would argue that you would catch and resolve this issue, if you are auditing your Docker images. (And you should be always doing your due diligence and checking your base and resulting images. ) So yes… you don’t really need virtualenvs, but you can also use them if you are concerned that you might be getting conflicting packages.

Volume maps

Graham was right about the adding volume mapping in the Dockerfile being problematic. You should not define volume mounts in your Dockerfile, since they create extra files with sudo-like permissions on the host (see /var). In your own datacentre that isn’t a problem. A multi-tenant cloud provider like OpenShift, would disallow you to create those files. The documentation argument I provided is not all that useful, since you can document the mountpoints in the README that you would provide with the Docker image.

Base Images

Base images are hard to get right. And there is a lot of debate whether or not to use tooling instead of base images. Graham says his warpdrive tool will do that sort of a thing. At work we build out our own tooling for building “standard” service Dockerfiles, and that just add another level of abstraction. I prefer base images since it while not ideal, provides less levels of abstractions that can get in the way when you’re debugging your Dockerfile setup. But your mileage may vary here.

So yes, good base images are hard. Try not to build your own unless you find it really useful and you have a great base to work from.

Installing GCC/Build Tools

In an ideal world one ought not have to include GCC, Python dev headers and so on. Yes, one can pip install using wheels, but that doesn’t always work out.

Dockerfiles

Formatting of the RUN command. This is not one of Graham’s points, but it did come up. Yes, you should format the RUN commands, with a line for each command and using a \ line continuation for readability. My slide didn’t have enough physical space to do so. My Rookeries example does a better job of this.

Running as Root

Graham is right, you should not run containerized apps as root. That is a bad security practise that can lead to an attacker compromising your Docker host via a privileged account on your Docker container. Again a bad example on my part, I should of added a USER command and dropped the VOLUME line, or maybe rethought the use of an example.

UWSGI and the HTTP flag

No, you don’t need it and you should use the UWSGI protocol if you put an NGINX container before your WSGI container. I left the flag in to make sure the example Dockerfile was runnable. My bad on trying to get a good illustrative example, but it wouldn’t be a good idea in production unless you feel comfortable exposing UWSGI to the direct HTTP traffic.

Personally I’m not a fan of mod_wsgi + Apache, but Graham did point out he created mod_wsgi-express to simplify your life. If we continue to use Apache + mod_wsgi at work, then I’ll try to get us to use mod_wsgi-express too.

Final Thoughts

Anyways, I hope got everything right. Thank you for reading all the way to the end! 🙂

See You at PyCon US 2016!

Posted on by

If you’re wondering why I’ve been so quiet these past few weeks, it is because I’ve been busy preparing to go to PyCon US in Portland this year!

I am very excited not only to be attending, but I will be giving a talk at PyCon US this year! I will be talking about Dockerizing Python microservices, and some of the lessons we’ve learned along the way at work. My talk will be on the first day (Monday May 30th) at 3:15-3:45 PM (PST). Videos of the all PyCon talks should be available a few days after the talk.

Huge thanks to everyone at my workplace, Points, for making this possible for me!

Finally I will be around in Portland for a few days after the sprints as well. I have never been to Portland, so I want to check out some of the sights around there. Let me know via Twitter or email if you want to meetup with me while I’m there. 🙂

Switching to systemd

Posted on by

I recently upgraded my version of Ubuntu to 15.04. In the process, I found out that my init system had changed from Upstart to systemd. While having something as fundamental as the init system management was a bit annoying, it isn’t as bad as some folks are making it out on the Web. Here are some of the things I learned as worked with systemd in fixing my CouchDB server. Part of this is based on this excellent guide on using systemd for Upstart users. The other part is just experimentation on my part.

Checking the Status of All Services

sudo systemctl status

Hint: feel to grep through the output to find anything

Sidenote

At first I could not find the CouchDB service. I had to uninstall the package and purge the packages and then reinstall it:

sudo aptitude purge couchdb
sudo aptitude install couchdb

After that the service showed up in the systemd services, rather than just having an Upstart service.

Checking the Status of a Service

sudo systemctl status $MY_SERVICE

Starting and Stopping a Service

sudo systemctl [start|stop|restart] $MY_SERVICE

Logging Service Activity

This was an interesting thing in systemd. Normally one has to look at the Upstart log at /var/log/upstart/$JOB.log. systemd provides its own logging mechanism, so to see the log of a service you have to use the journalctl utility.

sudo journalctl -u $MY_SERVICE

The output behaves just like one would expect from less.

Overall Impressions

Once I got over my initial head scratching of how to use systemd, it was not bad. Rather it feels different, but not in a bad way honestly. I might look into this more closely and see if I prefer using something like systemd over supervisord for controlling even WSGI apps.

The one disappointing thing that I discovered related to my experiments with systemd, or rather specifically with CouchDB in Ubuntu 15.04. There doesn’t seem to be a way to configure CouchDB to have admin users with passwords for some reason I can’t quite fathom. In the meantime, I guess I’ll have to stick to running CouchDB from a Docker container until I can resolve the issue natively.

Giving a Talk at PyCon Canada 2015

Posted on by

I am super excited to announce that I will be doing a talk at PyCon Canada this year! I will be talking about migrating from using Fabric to deploy my WSGI app (Rookeries) to using a combination of Invoke and Ansible. PyCon Canada will be happening in Toronto at the University of Toronto campus Saturday November 7 to Sunday November 8, 2015. My talk will on the Sunday at 3:45-4:15 PM. Videos of the talks should be available about a day or two after the talk. I look forward to seeing everyone there!

More info on my talk.

Also I plan on being at the Sprints the following Monday as well.

Fixing SPDX Expression Warning in package.json

Posted on by

If you ever run into the following warning when installing your NPM package:

npm WARN package.json rookeries-api-client-wrapper@0.4.9 license should be a valid SPDX license expression

That means you have an improper a name for the specified license in your package.json. So what are valid values for licenses? Well… here is the SPDX list of licenses

Command-line JSON Formatting with jq

Posted on by

About 2 or 3 months ago, when testing a deployment of a microservice at work
with Eric, our head Ops admin, we were looking at the JSON output of one of
the REST endpoints. Rather than looking at the raw output from curl, I
piped the output through JSON tool in the Python standard library:

$ curl -X GET -s http://rookeries.org/status | python -m json.tool
{
    "app": "rookeries",
    "version": "0.4.9"
}

(I will give you a couple of examples based on calls to Rookeries, rather than
the actual service call, since that API isn’t available publicly yet.)

Eric seeing that suggested I try out a utility that he uses: jq which is
great for formatting and querying JSON.

Installing jq

jq is a utility in C, and fortunately there are binary packages available for
it in the latest Ubuntu LTS (14.10). Installing it via aptitude (or apt-get):

$ sudo aptitude install jq

Pretty Printing JSON

The simplest use case for jq is to simply pretty print JSON output. This is
done by piping the result of a CURL command to jq with the parameter ‘.’:

$ curl -X GET -s http://status.bitbucket.org/api/v2/status.json | jq .
{
  "status": {
    "description": "All Systems Operational",
    "indicator": "none"
  },
  "page": {
    "updated_at": "2015-09-03T08:03:55.275Z",
    "url": "http://status.bitbucket.org",
    "name": "Atlassian Bitbucket",
    "id": "bqlf8qjztdtr"
  }
}

Formatting JSON in jq

It is also possible to format the output of the JSON to display only relevant
information. For instance if I want to find out the status of the components
that make up Bitbucket I can do the following:

$ curl -X GET -s http://status.bitbucket.org/api/v2/components.json | jq .

That however will give me a whole lot of extra data that I might not want. So
instead I might to narrow down and re-format the JSON data to something more
manageable with some jq magic:

$ curl -X GET -s http://status.bitbucket.org/api/v2/components.json | \ 
jq '{src: .page, components: [.components[] | \
    {id: .id, name: .name, status: .status}]}'

{
  "components": [
    {
      "status": "operational",
      "name": "Website",
      "id": "g0lfj4sv2fhf"
    },
    {
      "status": "operational",
      "name": "API",
      "id": "k0x2yw1435v7"
    },
    {
      "status": "operational",
      "name": "SSH",
      "id": "qmh4tj8h5kbn"
    },
    {
      "status": "operational",
      "name": "Git via HTTPS",
      "id": "c1qmcrcbc5zy"
    },
    {
      "status": "operational",
      "name": "Mercurial via HTTPS",
      "id": "vmbzxbbjz05j"
    },
    {
      "status": "operational",
      "name": "Webhooks",
      "id": "rfzky0v13fbp"
    },
    {
      "status": "operational",
      "name": "Source downloads",
      "id": "28h8dvv2qfzw"
    }
  ],
  "src": {
    "updated_at": "2015-09-03T08:03:55.275Z",
    "url": "http://status.bitbucket.org",
    "name": "Atlassian Bitbucket",
    "id": "bqlf8qjztdtr"
  }
}

I won’t explain that particular string in detail. But it will basically
craft a new JSON object, and generate new filtered objects when iterating
over the old array. Overall jq is pretty neat and is extremely fast to work
with.

There is also a Python bindings library for jq. I am considering using it to
help with mapping JSON into Python objects. However I have not played around
with it long enough to know if the extra dependencies are worthwhile or
whether or not it will bring a lot of benefits to Rookeries.

Reference Links

Spring Cleaning for 2013

Posted on by

With Easter just around the corner and possibly spring coming shortly after–Canadians have to wait a bit longer for spring t0 properly arrive and winter to make her final exit–that it would make sense to update my blog.   Many things have changed in the past few weeks .  Like we have a new pope, Pope Francis, just in time for Easter.  (I’m not going to weigh in on my opinions of the decision of the Conclave, other than I have mixed feelings.  And each passing day does not ease my general feeling about unease.)  Some things have not changed.  Like most things in the world I guess.

With the slow coming of warmer weather, I have a good excuse for a bit of spring cleaning and growing myself.  In terms of spring cleaning, I have meant to really organize my activities and my surroundings.  Unfortunately since I had to make do without my laptop for a few weeks, that has not helped me get more things done.  Especially when it comes to dealing with my overflowing inbox.  Apologies for everyone expecting me to get back to them.  I’m getting there slowly.

I did get to play around with setting up Python on my hosting environment and with Clojure.  Clojure, while definitely useful still feels like an exercise in academics than industrial programming.  (Still one can write a full implementation of Snake/Nibbles in Clojure in under 100 lines of code?  Madness!)  Python on the other hand is too much fun to feel like work.  I considered using something like a static website generator like Nikola or benjen to port some of my websites.  But I think for kicks, I will go the route of using Flask and craft my own mini-site just because working with Python is a such a joy.

One unfortunately necessary bit of spring cleaning will be changing Linux distros again.  It seems that Canonical is doing a fair bit of wild experimentation nowadays.  Too wild and it smells like they are suffering from NIH (not invented here).  The idea to chuck out everyone’s hard work on replacing X with Wayland, with their own thing was just too much.  So it looks like I’m going back to openSUSE for good.  It is just a matter of when I get around to migrating all my systems over.  I have no real issue with Canonical doing what they want with their own distro Ubuntu.  I just don’t agree with the philosophy, and the needless experimentation, especially since I am quite happy with using a relatively standard KDE 4 desktop.

Hopefully once I finish all the spring cleaning I’ll get to finish up and show off some the projects I’ve been working on.

 

Penguins and Pythons, Oh My!

Posted on by

Note: I have been meaning to get this post up for a while now.  However since I am swamped with work in general, everything is delayed and taking longer than expected.  Sorry about that.

As I mentioned in an early post, I am working on building up my professional experience in Python.  It has been a few months–time sure does fly when you are busy as heck–so it makes sense for me to update you on my experiences with Python so far.

Living in Large in Python-landia

I always enjoyed writing Python scripts.  Nothing quite beats Python when you need to write a script to search-and-destroy empty files or automatically converting files.  Sure you can suffer through BASH scripting or BATCH for the poor souls stuck in Windows.  You could write a one-liner Perl script that you could never ever read again.  Or you could bring in a 800 pound gorilla into the mess by coding a full-on program in Java or C++.  But why would you want to, if you can write a fully functional script in Python?

Writing scripts is one thing and writing applications is another.  An full application contains a lot of complexity, structure and code, and initially I felt uncomfortable building a large web application in Python.  Turns out it is not as big of a deal as I thought.  First of all a lot of the code that I write in Java is boilerplate or object manipulation or setter/getter fetish.  All of that goes away in Python.  Also it is easy to simply use other people’s libraries for much of that other work, so the size of a Python application is considerably smaller.  With a bit of careful organization and good code modularization, most of the problems go away by itself.

I found the last sections of Learning Python the Hard Way by Zed A. Shaw, an excellent reference as a both a refresher in Python and working with complex Python applications.  Incidentally I found the book so good, I think I will use it as a textbook when teaching programming.

As I use Python more and more, the more I like it the way it is.  Yes, I even have gotten over the whole whitespaces and tabs issue that most people get hung up about in Python.  Besides the brevity and elegance of the language, how can you not like a language that:

  • Has a Zen statement built into it.  (Hint: run import this in a Python shell.)
  • Refuses requests to be anything that it isn’t. (Hint: run from __future__ import braces)
  • Links to a xkcd comic about Python!  (Hint: run import antigravity)

Using PIP and PyPi (Or Shopping at The Cheeseshop)

One of the fun things I recently discovered in Java with Maven, Ivy and Gradle was the concept of automatically downloading and installation of dependencies.  I loved it that you could list the types of libraries that you’re application and everything automatically installed during the first build.  Also having lived with Linux package management, I got used installing all my software via a package manager.  Unfortunately many of the packages in Ubuntu are not the latest and greatest libraries that I need.  Also building a Debian deb or RPM is an exercise in yak shaving.

Fortunately all those issues are addressed in Python.  For package management you have the PIP utility, that installs packages available in a large web repository of Python libraries and applications called PyPi [http://pypi.python.org/].  This combination lets you install packages with a simple <code>sudo pip install $AWESOME_PYTHON_PACKAGE</code>.  Pretty nifty, eh?

If you are wondering about the reference to “The Cheeseshop”.  Apparently PyPi (not to confuse with the Python implementation of Python called PyPy) used to be called “The Cheeseshop” in reference to the Monty Python sketch by the same title.  Fortunately finding software that you want in PyPi is much easier than shopping for cheese at the Cheeseshop.

Charming Code with PyCharm and PyDev

Python code like I mentioned is one of those rare languages where you do not need an IDE to use, outside of a text editor and a shell/console.  I could not imagine doing the same with Java or C++.  For anything more complicated than a small function you really do need a full-blown IDE like Eclipse.  However after working with my favourite text editors Kate and Vim, I realized that I was missing better text highlighting and tab completion when writing Python code.  A good IDE really does help and not hinder when hacking in Python too.

Being familiar with Eclipse, I started using PyDev, a great Eclipse plug-in for Python with Django and Google App Engine support.  I highly recommend it, if you already use and are familiar with Eclipse.  However even when using the Eclipse plugins for Git and Syntax colour themes, I found that Python on Eclipse is still sort of a second class citizen compared to Java.  (Not sure if PHP or C++ developers on Eclipse have it any better in that case though.)

I experimented with quite a few different IDEs, while each offered something nice none of them really stood out.  Then I met the nicest Python IDE out there: PyCharm.   Of all the IDEs I’ve looked at PyCharm has the nicest tools, themes and feels much nicer than any other IDE out there.  Yes, I know that PyCharm is proprietary and that I am breaking my own rules about using libre software.  However there isn’t any IDE high quality enough that does what PyCharm does, including refactor, style checking and really good Django support.  The Git support could be nicer, but for any complex Git operation I end up using a console anyways.  Still I was impressed enough with PyCharm, that I actually bought a license for myself pretty much on the spot.

Other IDEs I’ve looked at included:

Dancing to the Sound of Django

While working with Python is pretty sweet, for me the big question is whether or not I can potentially make a living using Python.  Since most of my expertise is in web or web-powered mobile applications, I was most interested in what Python can do in that area.  Hence my interest in Django.

After using Django for a while, I can confirm that it is an amazing framework.  Django has captured my imagination of what can be done with Python.  While it takes some time to get used to the framework, it makes building web applications easy and enjoyable.  A lot of the tedious and mundane work like building administrative forms, mapping models to database tables and input validation, are dealt with for you.  Django is very modular, so with a bit of effort various underlying technologies can be swapped in and out.  Django provides smart defaults, but allows you to easily override the parts that don’t make sense.  One example is that it is fairly easy to override the default ORM SQL that is auto-generated from a model, to match the model or database better.

Documentation can make or break one’s perception of a library or framework.  Django has awesome documentation, and so do many of the community-contributed apps and systems of Django.  Some the add-on components really add a lot like Taggit and South, which provide tagging and database migration to Django.  Also there are some great books on Django:

One of the best parts of Django that I’ve encountered is the community of Django developers.  Unlike some communities, Django seems to get developers who are knowledgeable, friendly and approachable.  By hanging out on the #django IRC channel and asking the right questions, the devs there were able to help me out and answer some of my questions faster than I could of possibly been able to find the answers myself.  Of course as with any open source project community, you need to ask smart questions.  There are also local Django groups.  In fact I am looking forward to going to my first Django Toronto meetup.

Django is a really cool web framework for Python.  The designers of Django, really put a lot of effort in making a great framework.  The documentation is top notch.  The community is amazing.  To top it all, it is named after an amazing and cool French-Belgian jazz guitarist Django Reinhardt.  If you are into jazz or swing music I highly recommend checking out his music.

Using Python in Anger

 All the theory in the world, does not help if you can not put theory into practise.  Same goes with the theory of using Python in a professional setting.  Coming from the Java EE and Android worlds, I do expect certain tools.  Now since Python is interpreted on the fly, one can skip all of the compilation and packaging nonsense of Java.  (Yes, I know that Python code gets compiled into bytecode on the first run.  Why can’t Java do the same?)  As for API documentation, Python is quite a bit different than Java.  However epydoc does great job of API documentation, Sphinx does an incredible job with making in-depth documentation using the reStructured Text format.  Currently for my projects I have used epydoc mostly, and I’ve built a script to run it in Linux.  I have yet to figure out how to make a pure Python automated script for it, and I might switch to Sphinx if my documentation becomes extensive enough.
Interestingly in a company that I interviewed at recently, also uses Python and namely Pylons/Pyramid for the backend.  Also they have some Django components to if I remember correctly.  In fact a few of the people involved commercially in Python use a combination of Pyramid/Pylons, Django and some Zope, depending on the job.

Leave-A-Note Portfolio Project

While I have learned a lot about Django in the past few months, I still would hesitate on quitting my day job and becoming a Django consultant.  😛  However I have started working on a portfolio project to highlight my Django, web frontend and Android development skills.  This project is a simple device-to-device pull messaging system called  Leave-A-Note licensed under the AGPL.  While I could of easily made this application in PHP, I decided to build the server-web application in Django.  (I’m also working on another web app, but I can’t reveal much about that project yet.)  I am still in the initial phase of development, so there isn’t much to see yet.  I might build a few of the projects in the Practical Django Projects before I head off on my own.
All in all I am quite excited by Django.  Hopefully with it, I can start adding in more Python development in my professional programming repetiore, if not working with Python outright.

Diving into the Deep End with Python

Posted on by

Where I Stand with Java

Most of my professional experience in software development has centered around Java.  That should be no surprise as I learnt Java in university, and all the trappings that go with it: Javadoc, the Java Standard API, Eclipse and JUnit.  My professional work exposed me to JSP, Servlets, SpringMVC, Struts, JDBC, TestNG, Ant and the Android SDK.  Sure I’ve used other languages such as PHP, Perl, Scheme (mmm… functional programming…), C and Prolog.  The mainstay of my experience is still Java.

All that said, even as I include more and more exotic features of Java as language (i.e. enumeration, annotations, reflection, etc.) and as a platform, I find myself more and more constrained by the language itself.  Sure I feel comfortable with it, and I’ve written code personally that spans tens of thousands of lines.  However I find a lot of the code is boilerplate code.  Heck thanks to JAXB, I can quickly describe a series of classes using a XML schema.  That is right, a XML schema, one of those awfully obscure XML trappings that only enterprise application developers really should ever care about (or those working with complex document-style data).  Also a large Java codebase is nearly unapproachable with anything less than a powerful IDE.  Thank goodness for Eclipse!  (I write this wearing an Eclipse branded fleece jacket.)

Java is a wonderful and powerful language with great tooling.  However it just seems so wrong to have so much complexity.  I fear that this has more to do with the design decisions in the language itself than anything else.

Enter the Competitor: Python

I also learnt Python in university but I ended up using it largely for building system administration scripts rather than a general programming tool.  Instead of fighting with unwieldly BASH or Batch scripts for admin work, I use Python.  And I have created some powerful scripts using Python for tools for hunting down empty directories and rogue files for automated renaming.  After fighting with Java to do my bidding, I felt the need to look for some better language when building my new hobby project.  Also my recent attempts at learning Qt and C++ at the same time, have made me want to first learn the platform and then the language.  This is where Python seemed to hit a sweet spot.

While talking with one of my old university friends, Dan D’Alimonte, the topic of learning Qt came up.  Instead of using C++ for his Qt apps, Dan used PyQt.  After looking at his code, I decided to experiment with PySide and lo and behold I made a Qt app myself!  On the web side of things I heard a lot of good things about the Django web framework.  Hence I started experimenting with and investigating Python, PySide and Django.  In addition to the simple can-I-do-this-with-that experiments, and playing around with the tutorials, I researched the tooling and performance of Python based applications.

I am impressed so far with what I have found out about Python.  Coding in Python is a breeze and enjoyable.  I can comfortably use any text editor to program in Python.  I found that PyDev makes Python development while using Eclipse easy.  Getting new packages for Python is simple, and I do not have to rely on a package maintainer.  The standard libraries in Python are on par with Java’s.  You can always extend Python with a C or Java binding.  Building a desktop app using PySide turned out incredibly easy, compared to the standard C++ way of doing things.  Django is proving to be a very capable, and apparently *gasp* fast web framework especially when run on the optimized PyPy version of Python.  There is even mention of Python being able to build web frontends, at least in the documentation.  And importantly for me, Python has various XML, database layer, testing, and other enterprise and mobile capable libraries that I can potentially use in my day-to-day work.

Now naturally the quirks of a language become more readily visible once you actually start working on a real-life project.  That is my next step.  Then I can figure out if Python could really replace Java as my go-to language.  I will keep you posted once I get further in the project I am working on.  In the meantime, I’ve collected a bunch of links related to my investigation with Python.  Enjoy!

Python Related Links

Hello Again Identi.ca!

Posted on by

It seems that all the cool kids are using Twitter nowadays.  Then again what passes as cool now sounds more like either being different for different sakes.  Or that you use or do things that are “cool” or trendy or interesting to the mass public at the moment.  Rarely the cool, trendy or different thing is actually interesting or innovative or world-changing as much as it something that is well marketed.

As for myself I’m on Twitter because there are a lot of people that are not on my favourite microblogging platform: Identi.ca!  With Twitter being a bit more accessible on my Android smartphone and the smaller crowd on Identi.ca, I sort of stopped using it for a bit.  However that will change.  I prefer Identi.ca because it is libre software and my data is not held hostage in some corporation’s database.  I’ve updated my Identi.ca handle, wired up my microblogging widgets back to Identi.ca (and Twitter).  Now I plan on using it more often, and pushing my blog updates through Identi.ca first and then on to the other three popular social media that I use.  If everything goes as planned y’all should be able to read this post in a few moments after I publish it… right now. 🙂