Keeping Track of Time – Part 1 – Recognizing the Problem

I have a confession to make.  Like many other software developers, my time estimates are seem to vary from real time.  Yes, giving accurate time estimates are a difficult task, especially ones that are over long extended periods of time.  Add on top of that time seems like an illusion at times, and you have a perfect storm for inaccuracy.  However one of the hallmarks of a good senior developer are good time estimates.  So what to do?

Well you have to fix that problem like any else.  First lets do some research on the problem:

From the looks of it, the problem consists of breaking down a project or a problem into reasonable amounts.  Then one can build time estimates on the design, implementation, integration and testing of the components.  Sounds easy, yes?  Not quite, but that is a skill that can be developed.  How?  Well track the components needed to perform a certain task, measure the time it takes to finish the task and finally do analysis on the results.  Sure you can do this by hand, with nothing more than a pen, paper and stopwatch.  However this is far too tedious and onerus when developing.  Humanity invented and built computers for tasks just like this.  Again after a bit of searching online I found the following tools:

Over the next couple of days and weeks, I will play with each tool and try to figure out which works for me.  Finally I will write about which worked out the best for me overall.

Ghosts in the RSS Feed

A few days back I noticed a new entry in my RSS for an old blog of mine: http://opensourcegamer.blogspot.ca/

I was really surprised, since I had left that blog and closed down account years ago.  I started that blog when still in university and before I could afford my own fancy WordPress installation.  Years later after moving my old content and closing down that site, I find a new post.  At first I wondered if someone had hacked into my account and started doing weird things.  However this would be not the case.

It seems that a certain gent (https://plus.google.com/106427604834859977346/posts) decided that they liked the name of my blog and run with it.  That I don’t mind.  In fact I wish him (or her) the best of luck!  However please note that I am not maintaining that blog, and all my focus on this site.  So don’t be surprised if you see said odd entry in your RSS.  Come to think of it, you want to hear more of me, you should subscribe to the RSS feed for this blog!

Penguins and Pythons, Oh My!

Note: I have been meaning to get this post up for a while now.  However since I am swamped with work in general, everything is delayed and taking longer than expected.  Sorry about that.

As I mentioned in an early post, I am working on building up my professional experience in Python.  It has been a few months–time sure does fly when you are busy as heck–so it makes sense for me to update you on my experiences with Python so far.

Living in Large in Python-landia

I always enjoyed writing Python scripts.  Nothing quite beats Python when you need to write a script to search-and-destroy empty files or automatically converting files.  Sure you can suffer through BASH scripting or BATCH for the poor souls stuck in Windows.  You could write a one-liner Perl script that you could never ever read again.  Or you could bring in a 800 pound gorilla into the mess by coding a full-on program in Java or C++.  But why would you want to, if you can write a fully functional script in Python?

Writing scripts is one thing and writing applications is another.  An full application contains a lot of complexity, structure and code, and initially I felt uncomfortable building a large web application in Python.  Turns out it is not as big of a deal as I thought.  First of all a lot of the code that I write in Java is boilerplate or object manipulation or setter/getter fetish.  All of that goes away in Python.  Also it is easy to simply use other people’s libraries for much of that other work, so the size of a Python application is considerably smaller.  With a bit of careful organization and good code modularization, most of the problems go away by itself.

I found the last sections of Learning Python the Hard Way by Zed A. Shaw, an excellent reference as a both a refresher in Python and working with complex Python applications.  Incidentally I found the book so good, I think I will use it as a textbook when teaching programming.

As I use Python more and more, the more I like it the way it is.  Yes, I even have gotten over the whole whitespaces and tabs issue that most people get hung up about in Python.  Besides the brevity and elegance of the language, how can you not like a language that:

  • Has a Zen statement built into it.  (Hint: run import this in a Python shell.)
  • Refuses requests to be anything that it isn’t. (Hint: run from __future__ import braces)
  • Links to a xkcd comic about Python!  (Hint: run import antigravity)

Using PIP and PyPi (Or Shopping at The Cheeseshop)

One of the fun things I recently discovered in Java with Maven, Ivy and Gradle was the concept of automatically downloading and installation of dependencies.  I loved it that you could list the types of libraries that you’re application and everything automatically installed during the first build.  Also having lived with Linux package management, I got used installing all my software via a package manager.  Unfortunately many of the packages in Ubuntu are not the latest and greatest libraries that I need.  Also building a Debian deb or RPM is an exercise in yak shaving.

Fortunately all those issues are addressed in Python.  For package management you have the PIP utility, that installs packages available in a large web repository of Python libraries and applications called PyPi [http://pypi.python.org/].  This combination lets you install packages with a simple <code>sudo pip install $AWESOME_PYTHON_PACKAGE</code>.  Pretty nifty, eh?

If you are wondering about the reference to “The Cheeseshop”.  Apparently PyPi (not to confuse with the Python implementation of Python called PyPy) used to be called “The Cheeseshop” in reference to the Monty Python sketch by the same title.  Fortunately finding software that you want in PyPi is much easier than shopping for cheese at the Cheeseshop.

Charming Code with PyCharm and PyDev

Python code like I mentioned is one of those rare languages where you do not need an IDE to use, outside of a text editor and a shell/console.  I could not imagine doing the same with Java or C++.  For anything more complicated than a small function you really do need a full-blown IDE like Eclipse.  However after working with my favourite text editors Kate and Vim, I realized that I was missing better text highlighting and tab completion when writing Python code.  A good IDE really does help and not hinder when hacking in Python too.

Being familiar with Eclipse, I started using PyDev, a great Eclipse plug-in for Python with Django and Google App Engine support.  I highly recommend it, if you already use and are familiar with Eclipse.  However even when using the Eclipse plugins for Git and Syntax colour themes, I found that Python on Eclipse is still sort of a second class citizen compared to Java.  (Not sure if PHP or C++ developers on Eclipse have it any better in that case though.)

I experimented with quite a few different IDEs, while each offered something nice none of them really stood out.  Then I met the nicest Python IDE out there: PyCharm.   Of all the IDEs I’ve looked at PyCharm has the nicest tools, themes and feels much nicer than any other IDE out there.  Yes, I know that PyCharm is proprietary and that I am breaking my own rules about using libre software.  However there isn’t any IDE high quality enough that does what PyCharm does, including refactor, style checking and really good Django support.  The Git support could be nicer, but for any complex Git operation I end up using a console anyways.  Still I was impressed enough with PyCharm, that I actually bought a license for myself pretty much on the spot.

Other IDEs I’ve looked at included:

Dancing to the Sound of Django

While working with Python is pretty sweet, for me the big question is whether or not I can potentially make a living using Python.  Since most of my expertise is in web or web-powered mobile applications, I was most interested in what Python can do in that area.  Hence my interest in Django.

After using Django for a while, I can confirm that it is an amazing framework.  Django has captured my imagination of what can be done with Python.  While it takes some time to get used to the framework, it makes building web applications easy and enjoyable.  A lot of the tedious and mundane work like building administrative forms, mapping models to database tables and input validation, are dealt with for you.  Django is very modular, so with a bit of effort various underlying technologies can be swapped in and out.  Django provides smart defaults, but allows you to easily override the parts that don’t make sense.  One example is that it is fairly easy to override the default ORM SQL that is auto-generated from a model, to match the model or database better.

Documentation can make or break one’s perception of a library or framework.  Django has awesome documentation, and so do many of the community-contributed apps and systems of Django.  Some the add-on components really add a lot like Taggit and South, which provide tagging and database migration to Django.  Also there are some great books on Django:

One of the best parts of Django that I’ve encountered is the community of Django developers.  Unlike some communities, Django seems to get developers who are knowledgeable, friendly and approachable.  By hanging out on the #django IRC channel and asking the right questions, the devs there were able to help me out and answer some of my questions faster than I could of possibly been able to find the answers myself.  Of course as with any open source project community, you need to ask smart questions.  There are also local Django groups.  In fact I am looking forward to going to my first Django Toronto meetup.

Django is a really cool web framework for Python.  The designers of Django, really put a lot of effort in making a great framework.  The documentation is top notch.  The community is amazing.  To top it all, it is named after an amazing and cool French-Belgian jazz guitarist Django Reinhardt.  If you are into jazz or swing music I highly recommend checking out his music.

Using Python in Anger

 All the theory in the world, does not help if you can not put theory into practise.  Same goes with the theory of using Python in a professional setting.  Coming from the Java EE and Android worlds, I do expect certain tools.  Now since Python is interpreted on the fly, one can skip all of the compilation and packaging nonsense of Java.  (Yes, I know that Python code gets compiled into bytecode on the first run.  Why can’t Java do the same?)  As for API documentation, Python is quite a bit different than Java.  However epydoc does great job of API documentation, Sphinx does an incredible job with making in-depth documentation using the reStructured Text format.  Currently for my projects I have used epydoc mostly, and I’ve built a script to run it in Linux.  I have yet to figure out how to make a pure Python automated script for it, and I might switch to Sphinx if my documentation becomes extensive enough.
Interestingly in a company that I interviewed at recently, also uses Python and namely Pylons/Pyramid for the backend.  Also they have some Django components to if I remember correctly.  In fact a few of the people involved commercially in Python use a combination of Pyramid/Pylons, Django and some Zope, depending on the job.

Leave-A-Note Portfolio Project

While I have learned a lot about Django in the past few months, I still would hesitate on quitting my day job and becoming a Django consultant.  😛  However I have started working on a portfolio project to highlight my Django, web frontend and Android development skills.  This project is a simple device-to-device pull messaging system called  Leave-A-Note licensed under the AGPL.  While I could of easily made this application in PHP, I decided to build the server-web application in Django.  (I’m also working on another web app, but I can’t reveal much about that project yet.)  I am still in the initial phase of development, so there isn’t much to see yet.  I might build a few of the projects in the Practical Django Projects before I head off on my own.
All in all I am quite excited by Django.  Hopefully with it, I can start adding in more Python development in my professional programming repetiore, if not working with Python outright.

Linux SATA HDD Issues with Samsung RF-711

OK, this is a quick post for anyone who is running into issues with using Linux on a Samsung RF-711.  Specifically this applies to Ubuntu Linux 12.04 64-bit.

Essentially what was happening, was that periodically my entire machine would freeze with the exception of anything currently running.  This kept on happening all the time, until I wasn’t able to even use the machine for minutes at a time.  Which is a real shame since this is a very powerful and nice laptop.  Essentially my SATA hardware kept on hiccing up and failing to handle DMA write requests:

> dmesg

… (output abbrevated for display purposes) …

[ 485.289702] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 485.289712] ata1.00: irq_stat 0x40000001
[ 485.289720] ata1.00: failed command: READ DMA EXT
[ 485.289742] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 485.289744] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 485.289747] ata1.00: status: { DRDY ERR }
[ 485.289750] ata1.00: error: { UNC }
[ 485.432598] ata1.00: configured for UDMA/133
[ 485.432629] ata1: EH complete
[ 490.745427] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 490.745437] ata1.00: irq_stat 0x40000001
[ 490.745444] ata1.00: failed command: READ DMA EXT
[ 490.745458] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 490.745460] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 490.745467] ata1.00: status: { DRDY ERR }
[ 490.745472] ata1.00: error: { UNC }
[ 490.887945] ata1.00: configured for UDMA/133
[ 490.887980] ata1: EH complete
[ 496.201083] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 496.201093] ata1.00: irq_stat 0x40000001
[ 496.201100] ata1.00: failed command: READ DMA EXT
[ 496.201113] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 496.201116] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 496.201122] ata1.00: status: { DRDY ERR }
[ 496.201134] ata1.00: error: { UNC }
[ 496.354593] ata1.00: configured for UDMA/133
[ 496.354618] ata1: EH complete
[ 501.667841] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 501.667850] ata1.00: irq_stat 0x40000001
[ 501.667857] ata1.00: failed command: READ DMA EXT
[ 501.667870] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 501.667873] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 501.667880] ata1.00: status: { DRDY ERR }
[ 501.667885] ata1.00: error: { UNC }
[ 501.810534] ata1.00: configured for UDMA/133
[ 501.810566] sd 0:0:0:0: [sda] Unhandled sense code
[ 501.810571] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 501.810588] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[ 501.810593] Descriptor sense data with sense descriptors (in hex):
[ 501.810596] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[ 501.810606] 1c a7 c6 c0 
[ 501.810611] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[ 501.810616] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 1c a7 c6 c0 00 00 08 00
[ 501.810627] end_request: I/O error, dev sda, sector 480757440
[ 501.810655] ata1: EH complete
[ 508.210222] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 508.210228] ata1.00: irq_stat 0x40000001
[ 508.210232] ata1.00: failed command: READ DMA EXT
[ 508.210240] ata1.00: cmd 25/00:08:c0:c6:a7/00:00:1c:00:00/e0 tag 0 dma 4096 in
[ 508.210241] res 51/40:08:c0:c6:a7/00:00:1c:00:00/e0 Emask 0x9 (media error)
[ 508.210245] ata1.00: status: { DRDY ERR }
[ 508.210248] ata1.00: error: { UNC }

So every save, every time something needed to get loaded into memory was terribly slow.  After a fair bit of searching I found that the issue related to the parameters being passed to the SATA drive where my Linux partition lives:

> sudo lshw

… (output abbrevated for display purposes) …

*-storage
 description: SATA controller
 product: 6 Series/C200 Series Chipset Family 6 port SATA AHCI Controller
 vendor: Intel Corporation
 physical id: 1f.2
 bus info: pci@0000:00:1f.2
 logical name: scsi0
 logical name: scsi2
 version: 04
 width: 32 bits
 clock: 66MHz
 capabilities: storage msi pm ahci_1.0 bus_master cap_list emulated
 configuration: driver=ahci latency=0
 resources: irq:41 ioport:e0b0(size=8) ioport:e0a0(size=4) ioport:e090(size=8) ioport:e080(size=4) ioport:e060(size=32) memory:f7606000-f76067ff
 *-disk
 description: ATA Disk
 product: SAMSUNG HN-M101M
 physical id: 0
 bus info: scsi@0:0.0.0
 logical name: /dev/sda
 version: 2AR1
 size: 931GiB (1TB)
 capabilities: partitioned partitioned:dos
 configuration: ansiversion=5

The nearest bug report that I can point to on Launchpad is: Bug #550559 – hdd problems, failed command: READ FPDMA QUEUED

The solution that worked for me, and seems to be holding is to:
  • Turn off SATA GEN3 support in the BIOS.
  • Set the AHCI to manual and then disable it.
  • Add the following kernel parameter to your grub config: libata.force=X:noncq You will need to edit your /etc/default/grub file as I have below. Copy the /etc/default/grub.original file if you don’t have the a grub file already.
  • Rebuild your GRUB and initramfs with: sudo update-initramfs && sudo update-grub2
Sample grub configuration file:

# If you change this file, run ‘update-grub’ afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n ‘Simple configuration’

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash libata.force=1:noncq”
GRUB_CMDLINE_LINUX=”libata.force=1:noncq”

Anyways, this took me a while to fix.  So I’m posting in hopes that it’ll be help out someone else.