Django on Heroku: A Reader's Companion

I recently setup a fresh VirtualBox Debian Linux system. Within that context, I then walked through the Heroku tutorial on Getting Started with Django on Heroku. I ran into a number of errors along the way. Most of them took a few minutes (or an hour) of surfing to find a solution. If you've found this page, perhaps you hit one of those same error messages, and so my solutions will save you a little time.

I'll follow the sections of the current Django on Heroku tutorial here and just add the errors and solutions I encountered.

Prerequisites

There are a bunch of different Python installers out there: pip, distribute, setuptools, easy_install, etc. You'll need both pip and virtualenv... but virtualenv is listed specifically as a prereq at the start of the tutorial. According to the virtualenv page, the preferred way to get virtualenv is to use pip... but the best way to get pip is to use virtualenv. To resolve this, I decided to install pip globally... but pip requires distribute. Of course, the easiest way to get distribute is to use pip to install it (haha!)... but there is also this option where you use curl (or wget) to download an installation script. Once you have distribute, you can do something similar for pip.

Start a Django app inside a Virtualenv

It turns out the virtual environment needs a few things installed locally too...

Error: pg_config executable not found.

You need the postgresql package installed. (Use apt-get install postgresql to install it.)

Error: You need to install postgresql-server-dev-X.Y for building a server-side
extension or libpq-dev for building a client-side application.

Okay, so you also need the postgresql-server-dev-9.1 package (currently).

In file included from psycopg/psycopgmodule.c:27:0:
./psycopg/psycopg.h:30:20: fatal error: Python.h: No such file or directory
compilation terminated.

And the python-dev package too.

Hooray! At this point, I could get a Python app created in the virtualenv and run it. Things seemed downhill for me from here until...

Deploy to Heroku

Typing heroku create gave me:

!    Heroku client internal error.
getaddrinfo: Name or service not known (SocketError)

This error was coming from the excon gem. heroku login gave me the same error. Turns out I had a DNS problem. I entered 8.8.8.8 (Google's public DNS server) into /etc/resolve.conf and tried it again. Worked.

Then git push heroku master lead to:

Permission denied (publickey).
fatal: The remote end hung up unexpectedly

I needed to generate a public key with: ssh-keygen -t rsa By default, this puts the key into ~/.ssh/id_rsa.pub. You then need to register it with Heroku: heroku keys:add ~/.ssh/id_rsa.pub

At this point, I could execute the basic Django app both locally (with foreman start) and on Heroku.

Although my app wasn't actually using the database, I figured I'd better try syncing with it anyway...

Syncing the database

I wanted to run it locally first, so I needed to setup a local postgreSQL database. (The tips on the assignment page were very helpful here.) Also, you need to be postgre user in order to execute the psql command that starts the PostgreSQL console. (It took me a while to even track that command down.) So:

root@tinkVr:~# su postgres
postgres@tinkVr:/root$ psql
could not change directory to "/root"
psql (9.1.8)
Type "help" for help.

postgres=# CREATE USER django WITH CREATEDB PASSWORD 'djangoapp';
CREATE ROLE
postgres=# CREATE DATABASE a17 OWNER django;
CREATE DATABASE
postgres=# \q

You'll probably want to change the lowercase details (user, password, and database names) to something else. Note that you do not need to update your Django project's settings.py with these details because dj_database_url will pull these from from the DATABASE_URL environment variable. (If you log into Heroku's web interface and check your database settings, you can see what these values will actually be in the Heroku environment.) However, to run locally, you do need to set this variable appropriately. In my case, this translates to:

export DATABASE_URL=postgres://django:djangoapp@localhost/a17

typed at the command prompt.

At this point, I could successfully run python manage.py syncdb locally. However, heroku run python manage.py syncdb lead to a timeout. This happened even when just trying to open a simple shell on Heroku with heroku run bash. Heroku does offer some troubeshooting on this. It turns out port 5000 was being blocked by my ISP (the wired network in the Computer Science department on campus). I switched to the public UH wireless and I was able to connect.

The rest of tutorial went smoothly for me. (pip installs gunicorn for you.) Hopefully it goes smoothly for you too!

Next up: Fleshing out the Django app to actually do something...

Debian 7 on a VirtualBox

I'm currently taking a class on serious games. We've already covered Django and the nature of gamification. This week we are trying to deploy a simple Django project to Heroku, a cloud-based app hosting service. This proved fairly difficult, requiring that we get a long toolchain in place. (Details of that process to follow.)

This toolchain requires running scripts that download and install other required components. Whenever you're missing something, you get error messages that look like this:

  building 'psycopg2._psycopg' extension
  error: Unable to find vcvarsall.bat

You then have to figure out why you don't have vcvarsall.bat, or why it's not in the right place, and what you need to do to fix it.

Other students in the class reported that, since most of the required tools assume a unix-like environment, trying to get everything configured in Windows seemed to lead to even more problems. Also, project dependency managers like Ivy or pip always make me a little nervous. I don't really like the idea of downloading some dude's script from GitHub, running it, and having it download other arbitrary code from various points around the web and freely executing it all on my machine.

Given this context, I decided to create a Linux environment to work in. While I've dual-booted on a number of different machines before, I decided to try a virtual machine this time. There are a lot of advantages to this:

• No need to mess with your current partition configuration.
• Your can run the virtualized OS in a window, switching back and forth between the environments, even copying clipboard and file data between them.
• You can pause and save a running instance. (I've already used this feature: I paused a lengthy compilation/installation process so I could shutdown the real machine and then resumed everything again in the morning.)
• You can take a snapshot of the whole machine as a backup, which can be especially handy before you make any significant OS changes.
• You can copy the entire virtual OS environment to another machine. So, once you get your OS all configured, you're not bound to the original hardware.

For my virtual machine, I went with VirtualBox over VMware, since VirtualBox costs nothing and is open-source. Their provided documentation is pretty good too.

I then installed Debian 7 on it. (I actually did this twice, but here's what the process looked like the second time around. I'm assuming here that you know the basics of installing a Linux system.)

Due to previous Linux endeavors, I already had an ISO of the last stable Debian release (6.0/"squeeze") handy on a CD. (If I was doing this from scratch, I would grab a "testing" ISO instead.) I created a VirtualBox machine with a 12.0GB hard drive. (I hear this the only parameter you can't change once you create the machine.) Although it's a little slower, I went with a dynamically-sized file to save space on my host machine. I gave the machine 2GB of RAM and 1 CPU. (I'm used to running Linux on ancient hardware, so this is more than most of my installations get.) When I first started the virtual machine, it booted off the CD and began the installation wizard.

Debian's default Graphical Install worked out fine. When asked for what subsystems to install, I unchecked everything (including Standard System Utilities), so I had the barest machine possible. The default setup reserved 500MB of my 12GB for a swap partition.

The "stable" release tends to get outdated. For example, squeeze gives you Python 2.6. The current 2.7.3 is available in the "testing" release, and that will be updated within about a week of any new Python updates. So, once the installation was done, I edited my /etc/apt/sources.list to pull from "testing" rather than "squeeze". I then ran an apt-get dist-upgrade to switch over.

In upgrading, I got a new kernel. On rebooting, I found that my virtualbox guest services were not longer starting up properly. I would get this error:

AdditionsVBoxService error: VbglR3Init failed with rc=VERR_FILE_NOT_FOUND

At this point, the only obvious problem this caused is my mouse scrollwheel stopped working, but I knew it could cause other problems later on. I eventually learned that I needed to install the appropriate linux-header package. (While I love the idea of Linux, this is an example of why I don't actually use it everyday. You upgrade something that's supposed to be routine and suddenly something else breaks with some arcane error that requires an hour to debug, some config file hacking, and some new packages to track down. <sarcasm>Oh, you're obviously just missing the necessary the kernel headers. Duh!</sarcasm>)

For my desktop environment, I used: apt-get install lxde. This is pretty minimalist desktop, but I figured it should still be snappy in the virtual machine. So far, it is responsive. It looks crude coming to it from Windows 7, but I find I don't mind so much when I have Windows 7 running in the background, a single click away! When I get some time, I'm curious to see how well the latest KDE runs under these conditions.

I also installed openjdk7 and Eclipse (which wanted openjdk6 and a bunch of other stuff).

At this point, I had a working virtual Linux machine. Time to put it to work!

LXDE in maximized window with Windows taskbar readily available below.

Global Game Jam 2013: The Queen's Heart

I had fun hacking together a game called The Queen's Heart over the weekend as part of the Global Game Jam 2013. More info about the result can be found on its new project page.

In other unrelated news, I worked on picking up Clojure over the break, though that's lagged a bit now that another semester has started. I'm currently learning Git and Django (the Python-based web framework) as part of a class on serious games. All fun stuff!

Keyboard and Monitor Efficiency-Boosts

In online discussions, I've often seen two claims made about efficiency:

• Two monitors are better than one
• Master gurus work so fast because they never take fingers off the keyboard.

Over the past month or so, I've been exploring these for myself.

First, I bought an affordable 19" ViewSonic monitor for about $100 from Amazon. I also picked up a desk clamp, and so I was able to install it directly above my 15" laptop screen. The two are the same resolution, and, if I slide the monitor back back 3 or 4 inches, they're about the same size in my visual field. So I now have double the screen real-estate. I also installed a copy of UltraMon.

So far, the second monitor has been fun, but not all that useful. It's handy when I want documentation or my streaming radio up at the same time I'm working on some code. However, I find that two screenfulls of documents with white backgrounds are pretty bright at night, even though I cranked the brightness down pretty low. Also, even though the monitor is straight ahead of me, it makes my neck a little stiff if I work on it for too long. I guess I've gotten used to that slightly downward sightline of a laptop screen. Still, I don't regret the purchase and I'm fine with flipping the second monitor on whenever it's useful. I think (or at least hope) I'll gradually put it to more use over time.

The second notion I've been exploring is that keyboard-based computing is faster than mouse-based. This is often an argument for using emacs or (even more so) for vi. As mentioned previously, I've been playing with AutoHotKey. Here are a few of the interesting settings from my AutoHotKey script file:

#KeyHistory 0   ; don't record keypresses

I added this when I realized all my keystrokes--including the title of the current window--were being logged. Don't really need that info lying around.

~RAlt::Shift
~RCtrl::Shift

These turn right Alt and right Control into Shift+Alt and Shift+Ctrl. I've found this very handy with various macro and other keybindings.

This next setup was quite interesting:

CoordMode ToolTip, Screen   ;(at top of file)

SetCapsLockState AlwaysOff  ;;CAPS button does nothing alone

CapsLock & Enter::
 if GetKeyState("ScrollLock", "T") {
   ToolTip  ;turn it off
 }else {
   ToolTip NavMode, 0, 0
 }
 Send {ScrollLock}
 return
 
#If GetKeyState("ScrollLock", "T")
  i::Up
  j::Left
  k::Down
  ,::Down
  l::Right
  u::Send {PgUp}
  Space::Send {PgDn}
  h::Send {Home}
  `;::Send {End}
  a::Send {Alt Down}
  a UP::Send {Alt Up}
  s::Send {Shift Down}
  s UP::Send {Shift Up}
  d::Send {Ctrl Down}
  d UP::Send {Ctrl Up}
  f:: ;reserved for... mouse controls?
#If

CapsLock & i::Send {Up}
CapsLock & j::Send {Left}
CapsLock & k::Send {Down}
CapsLock & ,::Send {Down}
CapsLock & l::Send {Right}
CapsLock & u::Send {PgUp}
CapsLock & Space::Send {PgDn}
CapsLock & h::Send {Home}
CapsLock & `;::Send {End}

The idea I had here was that the different keyboard modes (ScrollLock, NumLock, CapsLock, and InsertMode) were going largely unused, but each could allow the keyboard to be completely remapped. My laptop keyboard doesn't even have a ScrollLock or NumLock key, and lacks any keyboard indicator lights. I thought I'd start with the least used--ScrollLock. Hitting CapsLock+Enter (a double-pinky-tap) puts me into ScrollLock and, since I don't have an indicator light, pops a very small "NavMode" tooltip in the upper left-corner of the screen. The homekeys (jkli,) become arrow keys, with h and ; as Home and End and U and Space as PageUp and PageDown. a, s, and d became alt, shift, and ctrl, respectively. You could of course take this further.

The homekeys-to-arrow-keys conversion would be handy enough to use without switching modes, so the second part means I can hold down CapsLock to turn the homekeys into arrow keys.

So, there we go: no reason to ever leave the homekeys again!

I found I hardly ever used this. I'm already trained on the arrow keys and the mouse, so that's where I go. If I'm moving my cursor far, I grab the mouse. If I'm going up a line or two, just a couple arrow-key taps are good enough. Sliding my right hand over 3 inches to use either of these wasn't costing me that much time, but trying to remember to use CapsLock instead was.

I still think it's a cool idea, though, which is why I'm sharing it here. Also, an approach like this would be very handy to implement a working number pad on my laptop, since it doesn't provide one.

In the end, I reverted to:

+CapsLock::CapsLock  ;Shift+CapsLock toggles CapsLock mode
CapsLock::Ctrl

which just turns CapsLock into Control.

I found I wasn't using this combo much a month ago, but this past couple days I've been exploring the fancier features of Eclipse. Ctrl+Space in particular is much handier with CapsLock as Ctrl.

Overall, my foray into keyboard efficiency has been disappointing. Supposedly the learning curve is worth it, but I doubt the extra few milliseconds are ever going to add up to the time it will take me to remap and retrain to do it. This old article suggests that my feelings on this are perhaps not so far off after all: that the speed of keyboarding is actually illusory.

This got me thinking of the QWERTY vs DVORAK debate. Supposedly, for experts, DVORAK is 10% faster (though this claim is contested). But, even if it is, I'm not hitting the QWERTY limit. That is, other people can type way faster on QWERTY than I can. I don't think it's primarily the keyboard layout that's slowing me down there; it my physical and mental processes that are too slow.

Bringing this back to the keyboard-vs-mouse debate, I see the same thing at play. I hardly ever mouse through a menu during my normal work, which I think is the main source of the "GUIs are slow" argument. I do use common keyboard shortcuts (like Ctrl+Z/X/C/V), etc, which are very fast when combined with mouse selection. When I don't use a keyboard combo, I generally hit a toolbar button. This is about as fast as a mouse-gesture. And even heading into a well-known and shallow menu doesn't really take that long. Striving for micro-improvments in this area is not really worth it to me. (Your mileage may vary, though, which is fine.)

This brings me in turn to my recent Eclipse experience. I'm realizing that Eclipse--with its Ctrl+Space and Alt+/ completions and various Content Assists and Code Templates--can drastically improve code generation times because it writes large sections of the code for you. I suspect that this is what may really be at work with older editors like emacs and vi: not that they are keyboard-only but that they are so full of useful tools. It's the intelligent and useful tools that make you faster, not the milliseconds you saved in selecting the tool.

So I've decide to give up on this idea of converting to entirely keyboard use. Instead, I'll be embracing my mouse and toolbar use. (For example, I mapped WinKey+ScrollUp to switch screens between monitors.) I'll see what I can do about speeding those up. I'll still use the keyboard and good bindings where appropriate, of course, but writing good macros is going to save me way more time than not moving my hand 3 inches over to the arrow keys.

I have one last insight on this, which is two-fold. First, keyboard sequences (such as Alt+F, S for Save on Windows or Ctrl+X, Ctrl+S on emacs) are about as fast as a more complicated finger-spreading key-combo (such as Ctrl+Alt+S). Secondly, menus actually provide and scaffold such combos. Alt+F gets you the File menu, and then every element there has a corresponding letter. You can get to any menu option in 2 or 3 keypresses--just like an Emacs key-combo. The difference is that the menu's always there to remind you of your options, and the menu path actually informs the keybinding. For example, in Eclipse, Ctrl+F11 is Run. I find this hard to remember. Why F11 and not some other function key? But consider: Alt+R, R, Enter. Not much longer, more compact on the keyboard, and memorable because it corresponds to: Run Menu, Run, (Execute). (The Enter is required because there is more than one R-bound menu option in the Run menu.) Sequences like this also give you more keybinding options at a factorial rate. For example, if you make Alt+J the start of all your Java-based macros, you get 26 three-key bindings (or many more, given all the other non-letter keys) with that short and relevant intro mnemonic.

Anyway, I'm sure I'll continue to think on this, but I'll be putting more effort now into intelligent macro design and Eclipse tool use than trying to speed-up my keyboard use. So, if you're still a coder putting your mouse to good use, you're not alone!

A quick fix to improve Python3 startup time

My web server is very low-end. Dating from the mid-90s, it has a 200Mhz Pentium and 96MB of RAM. It was running Debian 2.2 (potato), but I recently upgraded to the most recent Debian 6.0 (squeeze). I'm impressed that it even runs.

I also upgraded to Python3 in order to handle the recent overhaul of my Tamarin automated grading system. I compiled Python3 from source to do this, since Debian doesn't include it yet. They're still shipping Python 2.6.

Tamarin is little more than a bunch of CGI scripts. I expect it to run a little slowly on this machine, but, after the upgrade, any CGI request is taking about 4 seconds, which is fairly intolerable. Static webpages are still responsive enough, though. So I started up the Python3 interpreter... and waited. Yep, that's where the lag is. See for yourself:

ztomasze@tamarin:~$ time python3 -c 'pass'

real    0m2.493s
user    0m2.336s
sys     0m0.160s

Here, I'm just starting python3 to execute a single 'pass' statement that does nothing. This takes 2.5 seconds.

I read somewhere that not loading the local site libraries by using the -S option can give a performance boost. Since Tamarin uses only standard modules, I gave it a shot:

ztomasze@tamarin:~$ time python3 -S -c 'pass'

real    0m0.465s
user    0m0.392s
sys     0m0.072s

A 500% improvement! I even installed the default Python 2.6, just to compare times:

ztomasze@tamarin:~$ time python -c 'pass'

real    0m0.448s
user    0m0.348s
sys     0m0.060s
ztomasze@tamarin:~$ time python -S -c 'pass'

real    0m0.185s
user    0m0.148s
sys     0m0.036s

So Python3 is significantly slower for me than Python 2 was, but using the -S option at least gets me back to standard Python2 times.

This savings didn't really translate directly to improved CGI preformance though. Running two of my scripts from the command command line, I experienced the following:

                       status.py upload.py
original time          4 sec     6 sec
adding -S to #! line   3 sec     5 sec

A delay this long is still fairly intolerable. And I don't think the lag is inherent to Tamarin, since the delays weren't this long with Python2 and Debian 2.2 on the same machine.

I know I could probably shave off some more time for Tamarin by using FastCGI, SCGI, or mod_python. SCGI looks most useful to me given my existing codebase. Whenever I get some free time, I'll look into that.

Efficiency though Keyboard Shortcuts

One of the things I learned from the discussion of my recent Lisp posting was the value of delegating work to a good editor and the possible speed gain of good keyboard shortcuts. While I did not agree that the best path to these goals was necessarily through Emacs, I did decide to try to use my keyboard a bit more efficiently than I have been.

First, I reviewed the various keybindings already used by my OS, Windows 7. Hey, that Windows Logo key actually does have a few valuable uses! I also realized that I never use my function keys very much.

I also installed AutoHotKey and tried a few simple useful bindings, including:

• Capslock is now Ctrl. (Shift+Capslock acts as normal Capslock)
• Right Alt and Right Ctrl are now equal to Shift+Alt and Shift+Ctrl.
• Win+q quits a program (like Alt+F4) and Win+c opens a command prompt.
• The right-click menu button (AppsKey) is now a special function key. For example, I use it with various letters as shortcuts to certain directories when I'm in Windows Explorer.

I'm still working on actually using some of these on a regular basis, though. (Old habits die hard.) I'm also trying to use my alt keys with my thumbs without taking my fingers off the home keys, and using tab (replaced appropriately with spaces) more often when coding.

Anyway, it's all rather nerdy, but the possibilities are exciting. If you work on Windows, check out AutoHotKey. You may want to customize your system across all your different applications--especially if you start thinking about all the hundreds of possible key combos currently going unused on your keyboard!

Unit-testing a Python CGI script

This summer I'm overhauling Tamarin, my automated grading system. Under the hood, Tamarin is little more than a bunch of Python CGI scripts. However, as I overhaul it and convert it from Python 2 to 3, I also wanted to build a proper unit test framework for it.

It's been a dozen years or so since I last used Perl and CGI.pm, but I recall running my scripts on the command line and manually specifying key=value pairs. So, I was somewhat surprised to find no comparable way to test my CGI scripts in Python. The official Python cgi module documentation suggests the only way to test a CGI script is in a web server envirnoment. That's an unnecessarily complex environment for quick tests during development and precludes any simple separate unit tests.

In general, I'm not very impressed with the cgi module docs. In fact, browsing around revealed that there are a number of parameter options undocumented in the official docs.

Using this found information, I was able to build my own cgifactory module. Depending on the function called, it allows you to build a cgi object based on either a GET or POST query. For example:

  form = cgifactory.get(key1='value1', key2='v2')

If you then write your CGI script's main function to take an optional CGI object, you can easily build a CGI query, pass it to your script, and then run string matching on the (redirected) output produced by your script. Of course, most of your unit tests will probably be of component functions used by your script, but sometimes you want to test or run your script as a whole unit. cgifactory will help you there.

The cgifactory code is available here, where you'll always find the most recent version. The code itself is actually quite short; most of the file is documentation and doctests showing how to use it. I don't guarantee it's right, but it's worked for me so far. Hopefully it might be of use to someone else too! Feel free to copy, modify, and/or redistribute.

(Oh, and if you really need a command line version, it shouldn't be too hard to write a main that parse key=values pairs into a dictionary and then calls cgifactory.get(pairs) to build the CGI object.)