Category Archives: Computing

My thoughts and works in computing and programming.

Now a Professional Pythonista at Points!

Posted on by

I have been working for the past month as a Software Development Engineer at Points International.  While my role is not officially as a Python developer, a large portion of my work is building Python applications, services and libraries.  Also I get to develop in Java as well and maintain some very well engineered systems as well, so I get to deal with both worlds.  Even after a month, I am super excited to work at such a cool company and with awesome people.  It really feels like a bit of a dream job, in terms of what technology I get to use (Python, Linux desktops and distributed version control systems, w00t!) and the processes (yes Agile and proper software engineering totally works when done right).

But it is the people within the company that really makes it shine.  I get to be surrounded by smart, savvy, and welcoming coworkers, including a number of important and active Pythonistas that I look up to.  My team is just amazing, supportive, and I feel that in this short time span I’ve become a much better developer thanks to them.  Even on stressful days I feel motivated and excited to come to work and give it my all.  I feel incredibly lucky and fortunate to be at Points. 🙂

Distro Hopping

Posted on by

Sorry for the much delayed update, however this year has been an hectic and busy one. (New job, new house, lots of random unexpected events along the way, like two funerals and two weddings in a single month, etc. Long story.) Plus I really hoped to change blog platforms, but that is a story for another time.

Explaining the Journey

With so many things changing in my life, I decided to change up the Linux distribution I’m running. Now I have a large set of requirements being both a developer and gamer. I need a distribution that can handle Python, Java, Android and Qt Linux development. Also I want my distro to run Steam, and handle the Nvidia Optimus graphics card in my laptop, properly.

(Sidenote: A word to the wise, avoid Optimus cards as they are a pain to setup under Linux. I got mine because I naively assumed that all Nvidia cards are easily and nicely supported under Linux. Recently I heard that Nvidia promised to help the Nouveau devs to make the Optimus experience under Linux nicer. But I would not hold my breath to wait for things to get better soon.)

Long Story Short

The shortest version of the story: After doing a fair bit of distro hopping including using some uncommon distros, I am back to using Kubuntu.

Specifically the path I took was:
Kubuntu → openSUSE → Mageia → Debian → Linux Mint → Sabayon → Kubuntu u2192

The rationale behind all this? Well read on. 🙂

Kubuntu → openSUSE

After hearing about Canonical’s plans to use their own display manager “Mir” instead of “Wayland”, and experiencing random breakage with Kubuntu I decided to change distros. When I heard that the main dev behind Kubuntu was not going to be funded by Canonical, I decided it was time to jump ship.

I decided to retrace my steps, and try new versions of distros that I used in the past. Technically before I started using Kubuntu I ran on Gentoo Linux. But I was not about to go back to compiling and configuring everything on my system. So my first stop was openSUSE.

SuSE and now its community driven variant openSUSE, always has been a very slick distro in terms of supporting KDE.  The version I was running was no different. I was also encouraged by the large number of packages available including a nice setup for both Steam and bumblebee (this being the program that adds decent support for Nvidia Optimus under Linux).

openSUSE is a gorgeous distro overall, except for one very important issue… openSUSE feels like it was built for a corporate desktop. The number of PolicyKit warnings that I received whenever I tried to suspend and resume was surreal. While I am familiar with the lingo and ideas behind SELinux, AppArmour, etc, I could not for the life of my figure out how to get my laptop to resume and suspend without some silly PolicyKit message blocking me. openSUSE was not meeting my needs.

openSUSE → Mageia

With openSUSE failing me, I decided to go further in time to my original distro Mandrake/Mandriva. I found out that some Russian firm had bought out the French made Mandriva and as part of a general restructuring effort laid off some of the maintainers. These maintainers started their own version of Mandriva called Mageia. While the distro and its infrastructure is still fairly young, I was encouraged by the fact that some experienced maintainers were behind the project.

I was amazed with the amount of polish but into a budding community driven distro. I ran against some rough edges with Python support, but those were resolved with some help and new updates. I was impressed and I took my first steps to becoming a maintainer myself. The community was very receptive and welcoming. While I ended up using Mageia for weeks, I did not stay with the distro.

Why didn’t I stay with Mageia? I could not get bumblebee running on my machine. I could of fought some more, learn how to maintain a package and help build out the distro. But after some introspection, I realized that I simply do not have time contributing as a maintainer to a distro. There is a lot of work involved, and considering everything going on in my life right now, I needed to get a distro I could rely on and work with right now.

Magiea → Debian

Debian seemed like the logical choice for a stable Linux. The distro is entirely community driven, and has been around forever. So after a bit of haggling with the network installer, I managed to get a KDE desktop running on Debian. Debian definitely run on mature, stable software, which is perfect for someone running a server or managing a desktop configuration that has been around for years. Unfortunately the Linux desktop has only become very stable and usable in past while. Also the Debian community are sticklers when it comes to open source licenses, and how distributable
the software is legally. Unfortunately again, closed source firmware and other software makes things much more difficult. Getting my Broadcom wireless network card, and my Nvidia graphics chip working was just not happening.

Also I assumed that since Ubuntu worked so well, that Debian would be just as well setup from the get-go. I realize now how much work Canonical put into configuring their Debian base and smoothing all the wrinkles out. However I was not up for doing all that work myself, just to stay with Debian.

Debian → Linux Mint

Debian stayed installed on my laptop for a mere two days, before I got fed up with it. The next logical choice to avoid Ubuntu, but get some of the niceties of the platform was to try out Linux Mint. One of my good friends runs it and she enjoys using it thoroughly. I also watched and read some good reviews about the latest stable release of Linux Mint 15, and how much polish the devs put into the KDE desktop. I was intrigued, so I tried it out.

Linux Mint 15 definitely has a lot of polish. However nothing that spectacular that does not come standard to KDE. Except for the extra System Settings panel to handle PPAs (private Ubuntu repos), which is pretty darn cool. I did run into issues with trying to run packages originally meant for Ubuntu. There were slight and subtle incompatibilities, and I eventually gave up trying to fix things.

Linux Mint → Sabayon

By now I had run into a moment of madness. No good easy-to-use RPM based distros remained to try out. Fedora sounded too experimental for my liking. The Debian universe had been pretty much a let down. I debated using Netrunner, a KDE distro, by Blue Systems. (Blue Systems being that weird German company that somehow funds KDE development on Ubuntu, Linux Mint KDE and Netrunner. But no one has an idea how they fund themselves. Maybe by European Union funds, which seems to be the popular way to fund nebulous entities and projects in Europe.)

So I had a moment of madness, and despair brought on by no new leads while looking at potential distros on DistroWatch (http://www.distrowatch.com/). In that moment I decided to try a system not based on the traditional package systems. That left systems in the Arch or Gentoo families. Arch itself fell into the too much maintenance category. Gentoo did as well. Manjaro looks promising, but I’ll wait until it matures or fades way due to its small team. I tried Sabayon Linux, something I did not expect to do.

Sabayon Linux is definitely much nicer than Gentoo to maintain. Everything worked out of the box too. Except Sabayon felt very much like an early adopters hobbyist distro. An update or a new package installation, downloaded half the universe. My laptop ran faster… and ate its battery so quickly that it would just shutdown… randomly while running on battery. I could run Steam and my development environments, just never without worrying about my laptop suddenly powering off.

I realized I could not continue on like this…

Return to Kubuntu

Now I am back to running on Kubuntu, and everything just works well enough. I could of gone back to Mageia, and hoped that the upcoming release of Mageia 4 would of resolved most of my issues. Ultimately I went back to Kubuntu, since for right now it works well enough and meets my needs.

I work with Ubuntu at my new workplace, plus I support a couple of other Kubuntu machines running at home. I no longer use the tools that caused me grief when some libraries changed in Ubuntu. For better or worse, support for new applications or hardware is targeted at Ubuntu. Also it is a bit of a relief that Blue Systems stepped in and now funds development of Kubuntu. Canonical’s plans for transitioning to Mir, still do not affect me at least on my current version. Also this might change in the upcoming release, and I maybe stuck on this version of Kubuntu for a while.

Or things maybe change, maybe Canonical will change its mind and work with the Wayland community. Maybe Nvidia will fix up their terrible driver support due to market pressures. Or maybe I will have to move off to Mageia or Manjaro eventually. In the meantime I can be productive, and once things will calm down again, maybe I’ll go on another round of distro hopping.

Update (2013 October 18): Just upgraded to Kubuntu 13.10 yesterday!  I am encouraged by the news that the Kubuntu devs will push forward on using Wayland and support Kubuntu into the future.  So it looks like I will continue using and enjoying Kubuntu well into the future.  Now I’ll just need to learn how to package DEBs, and I’ll be able to help out occasionally too. 🙂

Spring Cleaning for 2013

Posted on by

With Easter just around the corner and possibly spring coming shortly after–Canadians have to wait a bit longer for spring t0 properly arrive and winter to make her final exit–that it would make sense to update my blog.   Many things have changed in the past few weeks .  Like we have a new pope, Pope Francis, just in time for Easter.  (I’m not going to weigh in on my opinions of the decision of the Conclave, other than I have mixed feelings.  And each passing day does not ease my general feeling about unease.)  Some things have not changed.  Like most things in the world I guess.

With the slow coming of warmer weather, I have a good excuse for a bit of spring cleaning and growing myself.  In terms of spring cleaning, I have meant to really organize my activities and my surroundings.  Unfortunately since I had to make do without my laptop for a few weeks, that has not helped me get more things done.  Especially when it comes to dealing with my overflowing inbox.  Apologies for everyone expecting me to get back to them.  I’m getting there slowly.

I did get to play around with setting up Python on my hosting environment and with Clojure.  Clojure, while definitely useful still feels like an exercise in academics than industrial programming.  (Still one can write a full implementation of Snake/Nibbles in Clojure in under 100 lines of code?  Madness!)  Python on the other hand is too much fun to feel like work.  I considered using something like a static website generator like Nikola or benjen to port some of my websites.  But I think for kicks, I will go the route of using Flask and craft my own mini-site just because working with Python is a such a joy.

One unfortunately necessary bit of spring cleaning will be changing Linux distros again.  It seems that Canonical is doing a fair bit of wild experimentation nowadays.  Too wild and it smells like they are suffering from NIH (not invented here).  The idea to chuck out everyone’s hard work on replacing X with Wayland, with their own thing was just too much.  So it looks like I’m going back to openSUSE for good.  It is just a matter of when I get around to migrating all my systems over.  I have no real issue with Canonical doing what they want with their own distro Ubuntu.  I just don’t agree with the philosophy, and the needless experimentation, especially since I am quite happy with using a relatively standard KDE 4 desktop.

Hopefully once I finish all the spring cleaning I’ll get to finish up and show off some the projects I’ve been working on.

 

PyCon and Beyond

Posted on by

Last weekend I went to the first ever PyCon Canada.  What an incredible event!  I met so many friendly, amazing, smart and talented people.  I learned so many new things, that essentially my knowledge of Python, and web technologies practically jumped to the next level over the course of 2 days.  The entire event left so inspired, that I’ve been hacking on a Python web application that I hope to release into the wild sooner than later.  Another fun Django Toronto night followed, and I learned so much there too.  I really can not wait to try out all these new technologies.  I have not gotten in touch with everyone from PyCon and Django Toronto, that I would like to.  Just been so swamped.  But I promise to do so shortly.

In the meantime, I hinted at a Python based web application that I am working on.  I won’t go into the details of the site in this post, since I’d rather show it off than talk about it. 🙂  I plan on building it out using Flask, SQLAlchemy and Jinja2.  Currently I am working through those technologies to build a particular website, and hopefully mastering them as I work out the details.  More details will follow soon…

For those who missed out on PyCon CA 2012: the videos of the talk are already up!  Check them out!

Now Accepting Tips via Gittip

Posted on by

I know this is totally an artsy thing to do but… you can tip me now!  So if you find my blog entries or the code that I put up on Github useful, feel free to tip me for my efforts.  Thankful my day job at Bluerush pays for my day-to-day.  But I would appreciate even a minor tip or a tweet, if you find these things useful.  It is almost like buying me a coffee or a beer. 🙂

Cheers!

The Python Scene, Accepting Javascript, and Lots of REST

Posted on by

Life continues at a breakneck pace.  Despite my best laid plans, the surest way of getting things done at the moment is in a sort of out-order manner.  However I did want to share a brief update on my journey into the Python and web application development world.  Instead of separate long form articles, I will briefly touch upon various topics.  (Note that this post itself has been in my work queue for over 3 weeks.  Mea culpae.)

The Python Scene in Toronto: Django Toronto and PyCon

I’ve now gone to two Django Toronto meetups and really enjoyed meeting with the Python community in Toronto.  The majority of web application developers in Toronto, deal with Java, Ruby or PHP (and of course Javascript).  However the Python community is there and growing.  They are very welcoming, and I have enjoyed each of the Django Toronto meetups I have gone to.  Also I feel like everytime I go there, I learn a lot and come back a better (or at least better informed) developer.  At the last meetup, I gained a deeper insight into crafting REST web services.  Something that I immediately applied to my day job the next day.  The one funny thing about Django Toronto, is that while the Django framework is an amazing platform, many Python developers move away from it once they scale up.  That is reflected in the meetups, since the topic of different platforms such as Flask come up.

The most exciting thing that is happening in the Toronto area for Python developers is PyCon.  This will be the first year that PyCon comes to Toronto.  I will definitely be there and I have been preparing by working on some real world Python applications.  If you will be there, I’d love to meet up with you!

How I Stopped Worrying and Started to Like Javascript

Javascript has been a language and platform I have avoided as much as possible.  If I can I try to stick to fancy tricks that one can pull off HTML5 and CSS3.  However one can only go so far without adding scripting to a web application.  Plain old Javascript still is a pain in the neck in a browser environment.  In other environments, it might be different as a scripting language.  However in such cases, I usually reach for Python rather than Javascript.  In a web app environment most of my experience has been with Sencha ExtJS.  While ExtJS is a remarkably powerful and flexible platform, it is problematic.  Aside from the licensing confusion of the original versions of ExtJS, I found that ExtJS contains a lot of automagic.  Building a ExtJS application consists more of configuring a number of components, and getting them to do more or less what you want to do.  Finding the right configuration has resulted in many stressful hours for myself and many other developers that I know.  Given all that developing with Javascript has left a bad taste in my mouth, until recently.

Two main things have changed my mind about Javascript.  Aside from the fact any sort of more advanced web application requires a full-on Javascript powered client, Javascript is not an entirely evil language for the web.  Doug Crawford (of JSLint fame) makes a good claim in his book Javascript: The Good Parts, that Javascript has some nice parts.  Yes, there are some ugly parts and needs a lot of love in the coming version.  But it is quite good for what is essentially a Schema-like functional/prototype language with Java-ish syntax.  Especially considering how quickly it was created at Netscape, it is not a terrible language.  If you are not convinced, I highly recommend watching the video below:

 

 

The other thing that changed my mind about Javascript is jQuery.  After fighting with ExtJS, the minimalist approach to Javascript development espoused in jQuery is refreshing.  I essentially learned the basics of jQuery overnight, trying to firefight a work project where office politics had gone amok.  (Aside: My take on politics in general is avoid it.  Get stuff done, not argue about it.)  Yes you have to know what you are doing with jQuery, and you don’t get a fancy MVC/MVT/etc. framework with jQuery.  But if you don’t fear writing Javascript by hand, and you want a library that abstracts the cross-browser stupidity, then jQuery is the answer.  For the next big web application that I am currently writing I will definitely be using jQuery.

Lots of REST and JSON

As mentioned earlier in the post, I have been dealing with a lot of REST service and JSON.  It is an infinitely nicer and simpler manner to talk with a server application, than via SOAP and XML.  (XML has its place, but it has been overused and abused in the Java world.)  When working with Java I’ve worked with Jersey and SpringMVC for building REST services.  Spring in general just works better, aside from its crazy arcane configuration.  In Python I’ve started working with Flask to handle building REST services, which I find a lot lighter than Django that sort of thing.  Also JSON is an awesome idea.  More people should use it for more of their data interchange needs. 🙂

IntelliJ IDEA Makes It All Better

Not to sound like a promotional campaign (since I work in what essentially amounts to the advertising industry, it happens more often I’d like) but one of the best decisions I’ve made recently is to switch IDEs.  I used to swear by Eclipse as the be-all-end-all of development environments.  Then I discovered PyCharm for Python.  Soon after that I got to meet Jessamyn Smith at a Django Toronto meetup.  While were talking about the joys of switching away from Subversion to Git–Jessamyn wrote a great article about her own experiences of migrating to Git–she convinced me to look into IntelliJ IDEA as it had a better interface for managing Git operations.  She was pretty convincing, as that is my primary IDE nowadays.  No more mucking around and wasting time with Eclipse’s temperamental setup.  Things.  Just.  Work.  Meaning I can do work.

Hitting the Flask

Somethings die hard.  One of those things is my own insistence on having lots of control over my computing environment and development platforms.  This led me to using Linux late in high school.  After playing around with Django, and wanting to build my own applications I found myself hunting down various odd ways to get around Django’s defaults.  Do not get me wrong, Django has a ton of nice pre-build features and default that just work.  Unfortunately being a web application developer, I have my own experiences, expectations and assumptions.  They are not always right.  However I prefer frameworks that I can plug-and-play and give me a finer grain of control.  (Hence I prefer using Spring in my Java web apps.)  So I’ve discovered Flask, a great micro-framework for Python.  I like how it makes web programming easy, without making a whole wack-load of design assumptions.  It very much reminds of me of the best parts of Spring, and apparently it is very “Ruby on Rails” like.

Keeping Track of Time – Part 1 – Recognizing the Problem

Posted on by

I have a confession to make.  Like many other software developers, my time estimates are seem to vary from real time.  Yes, giving accurate time estimates are a difficult task, especially ones that are over long extended periods of time.  Add on top of that time seems like an illusion at times, and you have a perfect storm for inaccuracy.  However one of the hallmarks of a good senior developer are good time estimates.  So what to do?

Well you have to fix that problem like any else.  First lets do some research on the problem:

From the looks of it, the problem consists of breaking down a project or a problem into reasonable amounts.  Then one can build time estimates on the design, implementation, integration and testing of the components.  Sounds easy, yes?  Not quite, but that is a skill that can be developed.  How?  Well track the components needed to perform a certain task, measure the time it takes to finish the task and finally do analysis on the results.  Sure you can do this by hand, with nothing more than a pen, paper and stopwatch.  However this is far too tedious and onerus when developing.  Humanity invented and built computers for tasks just like this.  Again after a bit of searching online I found the following tools:

Over the next couple of days and weeks, I will play with each tool and try to figure out which works for me.  Finally I will write about which worked out the best for me overall.

Penguins and Pythons, Oh My!

Posted on by

Note: I have been meaning to get this post up for a while now.  However since I am swamped with work in general, everything is delayed and taking longer than expected.  Sorry about that.

As I mentioned in an early post, I am working on building up my professional experience in Python.  It has been a few months–time sure does fly when you are busy as heck–so it makes sense for me to update you on my experiences with Python so far.

Living in Large in Python-landia

I always enjoyed writing Python scripts.  Nothing quite beats Python when you need to write a script to search-and-destroy empty files or automatically converting files.  Sure you can suffer through BASH scripting or BATCH for the poor souls stuck in Windows.  You could write a one-liner Perl script that you could never ever read again.  Or you could bring in a 800 pound gorilla into the mess by coding a full-on program in Java or C++.  But why would you want to, if you can write a fully functional script in Python?

Writing scripts is one thing and writing applications is another.  An full application contains a lot of complexity, structure and code, and initially I felt uncomfortable building a large web application in Python.  Turns out it is not as big of a deal as I thought.  First of all a lot of the code that I write in Java is boilerplate or object manipulation or setter/getter fetish.  All of that goes away in Python.  Also it is easy to simply use other people’s libraries for much of that other work, so the size of a Python application is considerably smaller.  With a bit of careful organization and good code modularization, most of the problems go away by itself.

I found the last sections of Learning Python the Hard Way by Zed A. Shaw, an excellent reference as a both a refresher in Python and working with complex Python applications.  Incidentally I found the book so good, I think I will use it as a textbook when teaching programming.

As I use Python more and more, the more I like it the way it is.  Yes, I even have gotten over the whole whitespaces and tabs issue that most people get hung up about in Python.  Besides the brevity and elegance of the language, how can you not like a language that:

  • Has a Zen statement built into it.  (Hint: run import this in a Python shell.)
  • Refuses requests to be anything that it isn’t. (Hint: run from __future__ import braces)
  • Links to a xkcd comic about Python!  (Hint: run import antigravity)

Using PIP and PyPi (Or Shopping at The Cheeseshop)

One of the fun things I recently discovered in Java with Maven, Ivy and Gradle was the concept of automatically downloading and installation of dependencies.  I loved it that you could list the types of libraries that you’re application and everything automatically installed during the first build.  Also having lived with Linux package management, I got used installing all my software via a package manager.  Unfortunately many of the packages in Ubuntu are not the latest and greatest libraries that I need.  Also building a Debian deb or RPM is an exercise in yak shaving.

Fortunately all those issues are addressed in Python.  For package management you have the PIP utility, that installs packages available in a large web repository of Python libraries and applications called PyPi [http://pypi.python.org/].  This combination lets you install packages with a simple <code>sudo pip install $AWESOME_PYTHON_PACKAGE</code>.  Pretty nifty, eh?

If you are wondering about the reference to “The Cheeseshop”.  Apparently PyPi (not to confuse with the Python implementation of Python called PyPy) used to be called “The Cheeseshop” in reference to the Monty Python sketch by the same title.  Fortunately finding software that you want in PyPi is much easier than shopping for cheese at the Cheeseshop.

Charming Code with PyCharm and PyDev

Python code like I mentioned is one of those rare languages where you do not need an IDE to use, outside of a text editor and a shell/console.  I could not imagine doing the same with Java or C++.  For anything more complicated than a small function you really do need a full-blown IDE like Eclipse.  However after working with my favourite text editors Kate and Vim, I realized that I was missing better text highlighting and tab completion when writing Python code.  A good IDE really does help and not hinder when hacking in Python too.

Being familiar with Eclipse, I started using PyDev, a great Eclipse plug-in for Python with Django and Google App Engine support.  I highly recommend it, if you already use and are familiar with Eclipse.  However even when using the Eclipse plugins for Git and Syntax colour themes, I found that Python on Eclipse is still sort of a second class citizen compared to Java.  (Not sure if PHP or C++ developers on Eclipse have it any better in that case though.)

I experimented with quite a few different IDEs, while each offered something nice none of them really stood out.  Then I met the nicest Python IDE out there: PyCharm.   Of all the IDEs I’ve looked at PyCharm has the nicest tools, themes and feels much nicer than any other IDE out there.  Yes, I know that PyCharm is proprietary and that I am breaking my own rules about using libre software.  However there isn’t any IDE high quality enough that does what PyCharm does, including refactor, style checking and really good Django support.  The Git support could be nicer, but for any complex Git operation I end up using a console anyways.  Still I was impressed enough with PyCharm, that I actually bought a license for myself pretty much on the spot.

Other IDEs I’ve looked at included:

Dancing to the Sound of Django

While working with Python is pretty sweet, for me the big question is whether or not I can potentially make a living using Python.  Since most of my expertise is in web or web-powered mobile applications, I was most interested in what Python can do in that area.  Hence my interest in Django.

After using Django for a while, I can confirm that it is an amazing framework.  Django has captured my imagination of what can be done with Python.  While it takes some time to get used to the framework, it makes building web applications easy and enjoyable.  A lot of the tedious and mundane work like building administrative forms, mapping models to database tables and input validation, are dealt with for you.  Django is very modular, so with a bit of effort various underlying technologies can be swapped in and out.  Django provides smart defaults, but allows you to easily override the parts that don’t make sense.  One example is that it is fairly easy to override the default ORM SQL that is auto-generated from a model, to match the model or database better.

Documentation can make or break one’s perception of a library or framework.  Django has awesome documentation, and so do many of the community-contributed apps and systems of Django.  Some the add-on components really add a lot like Taggit and South, which provide tagging and database migration to Django.  Also there are some great books on Django:

One of the best parts of Django that I’ve encountered is the community of Django developers.  Unlike some communities, Django seems to get developers who are knowledgeable, friendly and approachable.  By hanging out on the #django IRC channel and asking the right questions, the devs there were able to help me out and answer some of my questions faster than I could of possibly been able to find the answers myself.  Of course as with any open source project community, you need to ask smart questions.  There are also local Django groups.  In fact I am looking forward to going to my first Django Toronto meetup.

Django is a really cool web framework for Python.  The designers of Django, really put a lot of effort in making a great framework.  The documentation is top notch.  The community is amazing.  To top it all, it is named after an amazing and cool French-Belgian jazz guitarist Django Reinhardt.  If you are into jazz or swing music I highly recommend checking out his music.

Using Python in Anger

 All the theory in the world, does not help if you can not put theory into practise.  Same goes with the theory of using Python in a professional setting.  Coming from the Java EE and Android worlds, I do expect certain tools.  Now since Python is interpreted on the fly, one can skip all of the compilation and packaging nonsense of Java.  (Yes, I know that Python code gets compiled into bytecode on the first run.  Why can’t Java do the same?)  As for API documentation, Python is quite a bit different than Java.  However epydoc does great job of API documentation, Sphinx does an incredible job with making in-depth documentation using the reStructured Text format.  Currently for my projects I have used epydoc mostly, and I’ve built a script to run it in Linux.  I have yet to figure out how to make a pure Python automated script for it, and I might switch to Sphinx if my documentation becomes extensive enough.
Interestingly in a company that I interviewed at recently, also uses Python and namely Pylons/Pyramid for the backend.  Also they have some Django components to if I remember correctly.  In fact a few of the people involved commercially in Python use a combination of Pyramid/Pylons, Django and some Zope, depending on the job.

Leave-A-Note Portfolio Project

While I have learned a lot about Django in the past few months, I still would hesitate on quitting my day job and becoming a Django consultant.  😛  However I have started working on a portfolio project to highlight my Django, web frontend and Android development skills.  This project is a simple device-to-device pull messaging system called  Leave-A-Note licensed under the AGPL.  While I could of easily made this application in PHP, I decided to build the server-web application in Django.  (I’m also working on another web app, but I can’t reveal much about that project yet.)  I am still in the initial phase of development, so there isn’t much to see yet.  I might build a few of the projects in the Practical Django Projects before I head off on my own.
All in all I am quite excited by Django.  Hopefully with it, I can start adding in more Python development in my professional programming repetiore, if not working with Python outright.

Linux SATA HDD Issues with Samsung RF-711

Posted on by

OK, this is a quick post for anyone who is running into issues with using Linux on a Samsung RF-711.  Specifically this applies to Ubuntu Linux 12.04 64-bit.

Essentially what was happening, was that periodically my entire machine would freeze with the exception of anything currently running.  This kept on happening all the time, until I wasn’t able to even use the machine for minutes at a time.  Which is a real shame since this is a very powerful and nice laptop.  Essentially my SATA hardware kept on hiccing up and failing to handle DMA write requests:

> dmesg

… (output abbrevated for display purposes) …

[ 485.289702] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 485.289712] ata1.00: irq_stat 0x40000001
[ 485.289720] ata1.00: failed command: READ DMA EXT
[ 485.289742] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 485.289744] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 485.289747] ata1.00: status: { DRDY ERR }
[ 485.289750] ata1.00: error: { UNC }
[ 485.432598] ata1.00: configured for UDMA/133
[ 485.432629] ata1: EH complete
[ 490.745427] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 490.745437] ata1.00: irq_stat 0x40000001
[ 490.745444] ata1.00: failed command: READ DMA EXT
[ 490.745458] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 490.745460] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 490.745467] ata1.00: status: { DRDY ERR }
[ 490.745472] ata1.00: error: { UNC }
[ 490.887945] ata1.00: configured for UDMA/133
[ 490.887980] ata1: EH complete
[ 496.201083] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 496.201093] ata1.00: irq_stat 0x40000001
[ 496.201100] ata1.00: failed command: READ DMA EXT
[ 496.201113] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 496.201116] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 496.201122] ata1.00: status: { DRDY ERR }
[ 496.201134] ata1.00: error: { UNC }
[ 496.354593] ata1.00: configured for UDMA/133
[ 496.354618] ata1: EH complete
[ 501.667841] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 501.667850] ata1.00: irq_stat 0x40000001
[ 501.667857] ata1.00: failed command: READ DMA EXT
[ 501.667870] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 501.667873] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 501.667880] ata1.00: status: { DRDY ERR }
[ 501.667885] ata1.00: error: { UNC }
[ 501.810534] ata1.00: configured for UDMA/133
[ 501.810566] sd 0:0:0:0: [sda] Unhandled sense code
[ 501.810571] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 501.810588] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[ 501.810593] Descriptor sense data with sense descriptors (in hex):
[ 501.810596] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[ 501.810606] 1c a7 c6 c0 
[ 501.810611] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[ 501.810616] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 1c a7 c6 c0 00 00 08 00
[ 501.810627] end_request: I/O error, dev sda, sector 480757440
[ 501.810655] ata1: EH complete
[ 508.210222] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 508.210228] ata1.00: irq_stat 0x40000001
[ 508.210232] ata1.00: failed command: READ DMA EXT
[ 508.210240] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 508.210241] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 508.210245] ata1.00: status: { DRDY ERR }
[ 508.210248] ata1.00: error: { UNC }

…

So every save, every time something needed to get loaded into memory was terribly slow.  After a fair bit of searching I found that the issue related to the parameters being passed to the SATA drive where my Linux partition lives:

> sudo lshw

… (output abbrevated for display purposes) …

*-storage
 description: SATA controller
 product: 6 Series/C200 Series Chipset Family 6 port SATA AHCI Controller
 vendor: Intel Corporation
 physical id: 1f.2
 bus info: pci@0000:00:1f.2
 logical name: scsi0
 logical name: scsi2
 version: 04
 width: 32 bits
 clock: 66MHz
 capabilities: storage msi pm ahci_1.0 bus_master cap_list emulated
 configuration: driver=ahci latency=0
 resources: irq:41 ioport:e0b0(size=8) ioport:e0a0(size=4) ioport:e090(size=8) ioport:e080(size=4) ioport:e060(size=32) memory:f7606000-f76067ff
 *-disk
 description: ATA Disk
 product: SAMSUNG HN-M101M
 physical id: 0
 bus info: scsi@0:0.0.0
 logical name: /dev/sda
 version: 2AR1
 size: 931GiB (1TB)
 capabilities: partitioned partitioned:dos
 configuration: ansiversion=5

The nearest bug report that I can point to on Launchpad is: Bug #550559 – hdd problems, failed command: READ FPDMA QUEUED

The solution that worked for me, and seems to be holding is to:
  • Turn off SATA GEN3 support in the BIOS.
  • Set the AHCI to manual and then disable it.
  • Add the following kernel parameter to your grub config: libata.force=X:noncq You will need to edit your /etc/default/grub file as I have below. Copy the /etc/default/grub.original file if you don’t have the a grub file already.
  • Rebuild your GRUB and initramfs with: sudo update-initramfs && sudo update-grub2
Sample grub configuration file:

# If you change this file, run ‘update-grub’ afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n ‘Simple configuration’

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash libata.force=1:noncq”
GRUB_CMDLINE_LINUX=”libata.force=1:noncq”

Anyways, this took me a while to fix.  So I’m posting in hopes that it’ll be help out someone else.