So much Writing and Coding… so Little Time

I must apologize to regular readers of my blog, that I have been slacking. I plan on writing a blog entry once a week, and even though this one is late, I am trying to keep my promises. At least one should keep promises to one self. And I apologize if this entry is a bit sparse. It is a bit “phoned in”, I guess.

Anyways, I have been keeping busy with my many projects. Because of this I feel like I am pulling myself in a bunch of directions at the same time. This feels counterproductive and progress is slow. But I’m not sure of a better way to progress forward. I fear concentrating on one project, will just make everything else fall behind. Basically what I’m saying is that I’m trying to show some real progress on my open source projects, my not-so-open-source product, my writing, and bunch of other things… but yeah… it may be a while before I have anything substantial to show. Sorry.

Don’t even ask about all my emails… it is all a bit overwhelming. I will try to get back to everyone… but there really is so much to do. And only one of me… sigh…

In the meantime, I will try to keep up with the blogging. Maybe about some tips and tricks, but no real long articles for now. At least until I feel things get more manageable.

Anyways, take care and see you next week!

PyCon CA 2016 – The Art of Writing Wargames in Python with Kivy

Intro

Last year in November, I presented a talk about Kivy at PyCon Canada 2016. I talked about my experiences working with Kivy in the context of writing a strategy game. I discussed the various UI frameworks available to a Python developer, and why Kivy is currently the best choices for UIs in Python. Especially if you are interested in making a cross-platform app or game.

Video

Links

Abstract

Many of us got our start in programming, by building games and simple apps. However creating interesting games and UIs in Python was not a simple task especially for multiple platforms. Kivy, a cross-platform Python UI app framework changes all that. This talk discusses the journey of working with Kivy to develop a moderately complex strategy game. It also contrasts to using other platforms

Reaction

Overall, the talk went better than I expected. The audience was engaged and interested. There was a few technical difficulties, namely I had to resort to using my phone as a wifi hotspot to run my presentation rather than Ryerson University’s wifi. Also there was a noticeable delay between when I changed slides on my presentation notes, and the projected display. This kind of delay is probably the only downside of using slides.com this time around. Having a local copy would of worked better, which I will do so in the future.

Also I wish I had more time to come up with a better demo for the presentation. However even after the sprints, the hex grid layout still plagues my Kivy app. In hindsight, I probably should of worked on a simpler app. Also some folks were turned off by the use of wargames, rather general games. But you know, you learn from your mistakes.

Finally I got quite a few questions, and people are genuinely interested in checking out Kivy. Which ultimately was what I was trying to achieve with this talk.

Encrypt All the Things!

This week has been a rough one for me, so I’ll keep this blog entry short.

However I am pleased to announce that after a bit of struggle, I have setup HTTPS on all my sites, including this one. Thanks to Let’s Encrypt and specifically the acme-nginx project to making that possible. Getting everything setup, and automated using Ansible took a bit of work, but in the end it was worth it.

In the future I am looking forward to using the new more secure TLS 1.3 standard once it is supported more widely by NGINX and various browsers.

Merry Christmas and Happy New Year 2017!

Hey Readers,

Sorry for the delay between posts. I went on an unannounced hiatus, just because of the busyness of last year. However I am back, and I will return to my regular posting schedule of once a week, most likely either Wednesday or Thursday morning.

Also I hope everyone had a Merry Christmas (or has if you’re celebrating the 12 days of Christmas), and I hope your New Year is happy, bright and full of promise!

— Dorian

Avoiding Blank PHP Responses after upgrading to Ubuntu 16.04, and php7.0-fpm

Earlier this week I finally made the plunge to upgrade my VPS to Ubuntu 16.04. With a minor hiccup surrounding supervisord (which I probably can avoid if I go the systemd route) not being enabled at boot, the upgrade was simple for both my WSGI and Node webapps.

I can not say the same thing about my WordPress/PHP installations. (Installations that I hope to transition off to Rookeries once that software becomes more stable.) It took me a few hours to track down and resolve the problems. Hence I am posting this article, to hopefully save someone else’s time when they do the same upgrade.

Upgrading to PHP 7.0

Ubuntu 16.04 makes the switch away from PHP 5 to PHP 7. So I had to switch to php7.0, php7.0-fpm, and php7.0-mysql from their PHP 5 equivalents. The location of the running UNIX socket has changed from /var/run/php5-fpm.sock to /var/run/php/php7.0-fpm.sock, as did the PID files.

Updating the PHP-FPM configuration

Running WordPress using FPM (Fast Process Manager) and NGINX, requires turning off the path translation in php.ini file. This can be done by uncommenting the line cgi.fix_pathinfo=0 found in the configuration file /etc/php/7.0/fpm/php.ini. Again these files have moved from the old location. After you’ve done this remember to restart the FPM service using the new systemd utilities: sudo systemctl restart php7.0-fpm.

Updating the NGINX configuration and Solving the Blank Response

This is the tricky part. After updating my NGINX configurations to the new UNIX socket path, and restarting NGINX, I found that I got blank PHP responses. Everything else worked, expect that any PHP page would not render. And not render by not rendering any content in the body of the responses. That led me down a few rabbit holes, and researching how to re-architecture my setup using Docker. Eventually I stumbled across a blog entry with the solution to the blank PHP response issue.

In a nutshell, with the NGINX upgrade one of the parameters needed for FastCGI went missing namely the fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; bit. Oddly this parameter appears in /etc/nginx/fastcgi.conf and not the /etc/nginx/fastcgi_params file that I normally include in my NGINX configs. Anyways after adding this line and restarting NGINX using sudo systemctl restart nginx everything worked correctly. Below I’ve included a sample NGINX configuration that should work.

Sample NGINX Configuration /etc/nginx/site-enabled/example-site.conf

“`
server {
server_name .example-site.com;
index index.php;
root /srv/www/location_of_wordpress_install;
listen 80;

access_log      /var/logs/nginx/site-access.log;
error_log       /var/logs/nginx/site-error.log info;

location / {
    try_files       $uri $uri/ /index.php$args;
}

location ~ \.php$ {                        
    fastcgi_split_path_info     ^(.+\.php)(/.+)$;
    fastcgi_pass    unix:/var/run/php/php7.0-fpm.sock;
    fastcgi_index   index.php;
    fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
    include         fastcgi_params;
}

location ~* \.(js|css|png|jpg|jpeg|gof|ico)$ {
    expires         max;
    log_not_found   off;
}

}
“`

Selling Off Some Domains

I was going to write about why needing an app isn’t a good way to try get into business. (Hint: It is very expensive, if you can’t code it yourself.) However something else popped up, namely I am trying to sell off two domains: justcheckers.org and justcheckers.net that I own. These were for the justCheckers project, that I’m essentially shutting down. Anyways if you are interested in buying them off me, feel free to contact me.

I will say that the experience so far has been interesting. Thankfully there are some good resources for selling a domain or site. I’ve looked at two companies that handle the auction and transfer of these things Flippa and Sedo. Both are quite legitimate, but there were some bad reviews and some people claiming to have been scammed. I’m trying out Flippa first since it was founded by the fine folks at SitePoint, who I quite like for other reasons. Anyways, I’m hoping that I can find a good home for the domains.

See you at PyCon Canada 2016!

I’m looking forward to PyCon Canada 2016 that will be happening November 12-13 in Toronto. I submitted two talk proposals and I’m hoping that one of them gets accepted. But regardless I am looking forward to the conference. If you are at the conference, and you want to meet up just message me via Twitter @dorianpula. Also I plan coming out to the sprints that I’m hoping will be happening afterwards. See you there!

Rust, a New Toy Opening New Possibilities

I have not had a chance to blog in a while. Aside from the usual busyness of life, and the occasional bouts of illness, I have been distracted by a few new thing I’ve been learning: Rust, Kivy and Electron. I’ll write about Kivy and Electron in a future post. And a lot of that is centred around the upcoming product launches for Amber Penguin Software. But that is again for another post.

Rust

For the longest time, systems level programming (especially operating systems) have fascinated me. As part of that, I tried to learn the languages used to implement systems namely C and C++. While today I feel more comfortable with these languages, these still scare me with either their complexity (C++), their programming tools (gdb, gcc, autoconf, and minions) and their potential to do horrible things to your system if you are not careful. And bugs can be incredibly difficult to trace down and debug. So I while I have tried to code more C and C++, I still avoid them for these reasons.

Recently I started playing around with Rust, the new language developed partially by Mozillians. After trying to write a small Docker utility in Rust, and working through some koans for Rust, I have become smitten with the language. While some of the borrowing, references and types drive me a bit crazy, overall Rust is an amazing language. It brings the best ideas from Haskell, ML, Python and C and makes them very accessible. While the language is new and evolving the community has super active and creating lots of libraries for the language on crate.io.

Also I recommend listening to the New Rustacean podcast to learn Rust as well. It is not only informative, but very well executed by host Chris Krycho. So far I’ve listened to 10 episodes, and between the podcast, the koans and simply playing with Rust, I’ve learned a lot about Rust. In fact I feel more comfortable with Rust now then I have ever felt with C or C++.

Qt Bindings for Rust or Rather the Lack Of

On the topic C++, the one thing that keeps bringing me back to liar of C++ is the amazing platform that is Qt. However one on the most infuriating thing is that (aside from the limited QML), the only effective way to program Qt apps is using C++. Sure there is PySide and PyQt, with promises that PySide will actually support modern Qt5… some day… soon…

One of the things I hoped for Rust bindings for Qt to come quickly. Unfortunately it turns out that binding to Qt and C++ inside of Rust is really, really hard. There is an interesting project to develop some Qt C bindings for Rust, but it is really early in development and I had no luck using it. It probably is easier to bring in Rust to a C++ Qt project than the other way around.

In general UIs in Rust is a weak point for the language. Then again UI libraries are not the simplest thing to get off the ground, and it might be easier to rethink how we build them in general. Again this something I can get to in a future post.

PyCon US 2016 Talk – Pythons in a Container

At the end of May, I presented a talk at PyCon 2016, on using Docker with Python microservices. You can imagine the rush I felt getting to present on such a popular topic at such a large and important conference as PyCon! While it took me a while to recuperate after PyCon and Portland both of which were amazing. I would definitely do another talk at PyCon given the opportunity. Anyways I hope you enjoy watching the video of the talk! Below the video I also wrote about preparing for the talk, its reception, and a bit of the controversy that it stirred up below the fold. 🙂 (And I apologize for the lateness of this post, its been sitting in my backlog waiting to get finished for a few weeks now. 🙁 )

Video

Links

Abstract

Microservices and Docker are all the rage for developing scalable systems. But what challenges will you face when developing and deploying Python apps using Docker to production? This talk goes into the real-life lessons learned from creating, deploying and scaling Dockerized Python applications.

About the talk

Preparation for the Talk

PyCon talks definitely take quite a bit of time and effort to prepare. In my case, the talk took 3 major revisions before becoming the talk that I actually presented at PyCon. What started off as a intro to some of the concepts of Docker with some minor Python points, became more of a lessons learned targetted at intermediate to advanced developers. One of the things I wished I had (and I planned to but didn’t pull of) was to mention and thank my team for helping me preparing my talk. So thank you Kevin Qiu, Biniam Bekele, Yele Bonilla, and Gavin D’Mello for all your support, sitting me through three versions of my talk, and all the amazing feedback! I’ll make sure to include a slide with thanks next time.

Also I am very thankful for Jared Kerim from Mozilla, who presented at the local Python Toronto meetup about his team’s Docker setup. He was kind enough to let me use his example docker-django-template project. His example was also inspirational for my own docker-compose example using my CMS Rookeries. An example that I crafted and tested on the plane trip over to Portland. (After talking with other speakers, finishing up your presentation, notes and examples on the plane trip over is a proud traditional of PyCon and other conferences. :D)

Reception

Overall the reception of the talk was amazing! The talk turned out quite a crowd, in fact filling up most of the room. (I’m not sure of the capacity of the room but I estimate over 300 people attended). I was pretty nervous, but with the exception of a few stumbles, I think I pulled off the talk quite well. I really enjoyed some of the questions that were fielded during the Q&A session, and also privately afterwards. I wish could of answered some of the Docker Machine and Amazon ECS questions better, but I simply have not worked with both technologies long enough to give proper advice.

Controversy

The most surprising aspect of the talk was the controversy it stirred up. At the end of the Q&A you can hear some comments from a young lady about where I supposedly went horrbily wrong, and how there were tweets flying back and forth about it. I had turned off the notifications on my phone when I got up on stage, to avoid getting distracted. She persisted with telling (or trying to explain) what was wrong in the private gathering afterwards. Unfortunately she did not do a wonderful job of communicating, and I felt it took away time from others to ask their questions. It didn’t help her case that she admitted to being a novice at Docker. Please don’t that as an attendee, there are better ways to disagree and communicate that.

I later approached by a gentleman (thank you whoever you are), who mentioned I should go talk to the OpenShift guys since they had some concerns about my talk. News of the Twitter controversy worried me, because I hated the notion that I had gotten on stage and toled people to go and do the wrong thing. Especially when apparently I’m telling the opposite of what Glyph from Twisted said to do. After a brief chat (and a nice demo about their cool Kubernetes suite) from the OpenShift guys, I found out that Graham Dumpleton, the creator of mod_wsgi and who works on OpenShift had done a live tweeting commentary during my talk, where he disagreed with a few of my points. Long story short, eventually I was able to chat with Graham. He was a great sport and explained his points. Interestingly enough I had also talked with the folks at Docker. And they agreed with the points in my talk, and the logic behind my points. Essentially most of my points were based off the best practises they proposed.

Anyways I listed a few of Graham’s points with links to his blog posts (thanks again Graham!), and some of my quick thoughts on each one. A quick disclaimer about some of my points: the advice I gave worked for us in our datacentre, and that it might not work for others in other environments. It should work well, it might not be perfect, but it worked for us, and some of the folks at Mozilla. I gave a disclaimer at my other talk on a Ansible setup for WSGI apps at PyCon Canada, and I thought it was superfluous. But it turns out it is a useful thing to mention, and be explicit.

Errata

So the slide that caused a good portion of the controversy was the base image one. There I had provided an example Dockerfile on half the slide and discussed about base images and good Dockerfile practises, with points on the lower half. Now the example was meant as a toy and not necessarily complete. It is difficult, even impossible to present a well formated, perfect Dockerfile in that context. There is only so much room on a slide to fit both an illustrative example and some explanatory points. That is why I included links to some samples, that hopefuly did a better job of it.

Virtualenvs

Ah yes, the “enfant terrible” of my talk. 🙂 If you want to be controversial in your talk, mentioning something like this will get people’s attention. (Ironically, it was not my desire to stir up a controversy). Graham post a while back why you might want to use virtualenvs in your Dockerized app. It is a longish post, so I’ll give a shortened version. Basically when you base your image off some distro (say Ubuntu, Fedora or what not), there is a good chance of bringing in more Python packages in your system site packages than you expected. e.g. You’re building a Flask app, and the package maintainer included a version of Werkzeug in the base Python install, so now when you pip install Flask as part of your requirements you get the wrong version of Werkzeug.

And that is a valid point (with my example)… except if you use something like the official Python 2.7 base image… which installs just Python. I would argue that you would catch and resolve this issue, if you are auditing your Docker images. (And you should be always doing your due diligence and checking your base and resulting images. ) So yes… you don’t really need virtualenvs, but you can also use them if you are concerned that you might be getting conflicting packages.

Volume maps

Graham was right about the adding volume mapping in the Dockerfile being problematic. You should not define volume mounts in your Dockerfile, since they create extra files with sudo-like permissions on the host (see /var). In your own datacentre that isn’t a problem. A multi-tenant cloud provider like OpenShift, would disallow you to create those files. The documentation argument I provided is not all that useful, since you can document the mountpoints in the README that you would provide with the Docker image.

Base Images

Base images are hard to get right. And there is a lot of debate whether or not to use tooling instead of base images. Graham says his warpdrive tool will do that sort of a thing. At work we build out our own tooling for building “standard” service Dockerfiles, and that just add another level of abstraction. I prefer base images since it while not ideal, provides less levels of abstractions that can get in the way when you’re debugging your Dockerfile setup. But your mileage may vary here.

So yes, good base images are hard. Try not to build your own unless you find it really useful and you have a great base to work from.

Installing GCC/Build Tools

In an ideal world one ought not have to include GCC, Python dev headers and so on. Yes, one can pip install using wheels, but that doesn’t always work out.

Dockerfiles

Formatting of the RUN command. This is not one of Graham’s points, but it did come up. Yes, you should format the RUN commands, with a line for each command and using a \ line continuation for readability. My slide didn’t have enough physical space to do so. My Rookeries example does a better job of this.

Running as Root

Graham is right, you should not run containerized apps as root. That is a bad security practise that can lead to an attacker compromising your Docker host via a privileged account on your Docker container. Again a bad example on my part, I should of added a USER command and dropped the VOLUME line, or maybe rethought the use of an example.

UWSGI and the HTTP flag

No, you don’t need it and you should use the UWSGI protocol if you put an NGINX container before your WSGI container. I left the flag in to make sure the example Dockerfile was runnable. My bad on trying to get a good illustrative example, but it wouldn’t be a good idea in production unless you feel comfortable exposing UWSGI to the direct HTTP traffic.

Personally I’m not a fan of mod_wsgi + Apache, but Graham did point out he created mod_wsgi-express to simplify your life. If we continue to use Apache + mod_wsgi at work, then I’ll try to get us to use mod_wsgi-express too.

Final Thoughts

Anyways, I hope got everything right. Thank you for reading all the way to the end! 🙂