SourceTree: My new favorite Git GUI

I've started a new job, so I've been offline for a while. But I want to give a quick shout-out to Atlassian SourceTree.

I like Git, and I'm not afraid of the command line, but I'd rather use faster, more informative GUI tools if they exist. I've been using GitHub for Windows for my projects stored on GitHub. I learned about SourceTree at work and, now that I know it also integrates with GitHub, I've switched over at home as well.

GitHub for Windows is very simple, which is an advantage when you're just getting started with Git. But I find certain things can only be done through the GitHub web interface or on the command line. SourceTree, on the other hand, has a lot more power to it. As just one example, you can easily pick chunks and specific lines of a file to include as a commit.

If you like a GUI front-end to your Git, but you want a bit more power and elegance, check out SourceTree.

HabitMaster: Alpha release

HabitMaster is now live in the wild. You can see the new one-page HabitMaster project site that links to everything else you might want to see, including a running Heroku instance.

Overall, this deadline has been met with limited success. I'm calling this an alpha release because the HabitMaster achievements (in other words, the actual gamification parts!) are still unimplemented. It means HabitMaster is just a basic habit tracker right now, and still a little rough around the edges at that.

This has mostly been due to just slow progress. According to my time tracker, I averaged around 8 hours a week on ICS691 projects for most the semester. I stepped that up to averaging around 12 hours a week for the past month, but I still don't appear to have much to show for it. Yet it seems all of my implementation projects go this way: I continue to underestimate just how much code is required to do things.

I spent most of my time implementing back-end details, such as streak detection given a list of activities and a particular schedule type. I was excited to use a Python generator (basically a fancy iterator) to good effect there.

I also came up with a workable solution to handling polymorphism in database objects. Specifically, my habits link to a Schedule object, but there are two subtypes of Schedule: DaysOfWeekSchedule and IntervalSchedule. Grabbing habit.schedule would give me a row from the Schedule table, which was fairly useless. I ended up adding a cast() method to Schedule that would resolve that instance to the proper subtype for me.

Django's unittest framework was very handy. I didn't get into the UI unit tests that actually let you test requests and responses, though. I would like to explore that more. I'm still impressed with Django and its documentation.

I picked up more HTML5 tags, which has been enjoyable. I dig all the semantic tags they have now. Twitter Bootstrap and CSS design in general was a time drain, though. There was a lot of fidgeting required to get everything laid out right. I'm still a lousy graphical designer. CSS still trips me up on things that seem like they should be easy, such as define a DIV of a given relative width and then align a couple items to the left and one or two to the right within that DIV. And then have them all play nicely as the window resizes. I will say that Bootstrap helped compared to trying to do this from scratch in CSS, though.

GitHub (and to a lesser extent Heroku) has been an increasing pleasure to use. I put GitHub Pages and GitHub's wiki, issues, and milestone features to work on this project. I code alone so often that I've developed some lazy git practices, though. The main one is that my commits are based on work-session rather than issue. It doesn't affect anyone else, but it means I often commit things half-done or in a broken state. That is going to come back to haunt me if I ever have to rollback to a previous verison. I also have a lot yet to learn about merging and pulling with git. I ran into a bit of that while I was editing on Windows, Linux, and through the GitHub web interface today.

As you can probably tell, I enjoyed all the technical learning this project has afforded me.

Regarding the gamification aspects, I have this lesson to share: It takes a lot of time. Any gamification mechanics will be in addition to implementing the underlying service you want to provide. The gamification needs to be well-thought out, too. If possible, leveraging an existing framework could help cut down on that time... though it means you'll need to spend the time learning the framework and then being bound by its limitations.

The other thing that HabitMaster has clarified for me is the difference between gamification and a serious game. I see HabitMaster as an example of the former but not of the latter. To me, gamification means being inspired in your design by the fun and rewarding engagement we find in games--even when you're designing something that is clearly not a game. Because it's not a game, you can't just paste game mechanics onto your service. You need to think about the experience you're going for... which brings us to deeper concepts like mastery, autonomy, meaning, flow, engagement, reward, and the like. A serious game, on the other hand, is when you take something that is clearly a game and try to graft non-game goals or outcomes on to it.

HabitMaster will likely be dormant for a little while as I finish up some other pressing projects, but I do hope to resume development when I get some time. I'll also be starting a new teaching job in July. I wonder if I might bring some gamification to bear on course design there.

HabitMaster Progress

This past week or so I've been implementing my previously-mentioned Habit Master design.

So far, things are going slowly but smoothly. I decided to divide my design into three apps: habits, users, and rewards. I first defined the habit and schedule models, some unit tests for them, and got them setup for admin view access. I also defined a User model before I realized it would be a much better idea to reuse Django's existing auth system for that. I still have a users app as a good place to put the views and page templates associated with account creation, login, and logout. I learned more about how templates work, so I have a base page template I'm inheriting from. I've also sorted out how static files work for things like style sheets and html5shiv.js. I'll eventually get Twitter Bootstrap into that mix too, but I'm focusing on functionality first.

There have been no major obstacles or bugs so far (knock wood). It's just been slow due to the general learning curve. Though I feel like I had a comfortable conceptual overview thanks to earlier exercises this semester, I still had to skim through the tutorial again as I completed each step for this project. I've also been diving deeper into the Django documentation for things like the auth system, view decorators (a bit like Java annotations), and request object details. I think Django could have provided more built-in help with my login form, but I haven't dug into that yet. Overall, I still like Django and its documentation.

I've been reading the HTML 5 spec to pick up some of the many new tags and throw them into the mix too.

There have also been a few toolchain improvements along the way. For example, I learned that git add . doesn't remove deleted files from the repository. To handle this, I learned more about git add -u, which will update all modified and deleted files but not stage any new additions, and git add -A, which will properly stage all changes. I've also been improving my virtualbox linux environment. For example, I found that KDE's Kate editor does allow to keyboard binding customization after all: you just have to go to Settings -> Configure Shortcuts... instead of Settings -> Configure Kate... to find them.

Hopefully things continue to go well. It doesn't seem like I have much to show at this point, but it feels like I've touched all the major parts at this point--apps, settings, models, tests, views, templates, urls, static files, and deploying to Heroku--so the rest should hopefully just be fleshing out more of the same. The next two or three weeks will tell.

HabitMaster: Gamified Habit Formation

I'm starting on a new web-based gamification project. The current design is on GitHub.

This design has been evolving from its original broad concept. Some recent design additions include:

• Users can now record notes regarding each task completion, or even if they did not complete the task for a given day.
• Users can specify degrees of completion (green for "wholehearted" or yellow for "halfhearted"). However, to avoid discouraging those just starting out, this will be an advanced feature that is unlocked only once a habit has been firmly established.
• Social features have been given a bit more consideration, though they will still probably not be implemented as part of version 1.0.
• Fleshed out a number of specific details about the interface and flow between screens, including three new UI mockups.

Version 1.0 will include only the core features. I could evaluate the effectiveness of these gamification features by creating a second version with them stripped out. Specifically, this would omit the mechanics of streak tracking, leveling up a habit, badges, milestones, 1UPs, and social sharing. This would leave only the ability to enter habits, check them off each day, and review their history. I could then compare user adoption of the two systems: would users use the system longer and more frequently, have more successful habit formations, and enjoy the process more with these extra gamification features?

I'm looking forward to starting the implementation next week and learning some Django from scratch.

Makahiki Widget Development

My exploration of the Makahiki platform continued this week. I tried my hand at adding a few new widgets that could be inserted into a different pages. (Apologies to any general readers: this will be a very Makahiki-specific post.)

I started by following the Hello World tutorial. This was fairly painless and gave a nice overview of the files and functions that comprise a widget.

Next, I built a Group Scoreboard widget. At this point I wished for a bit more of a high-level overview of how widgets should work. I wasn't sure where the best place would be to add new data-gathering code: within the template HTML page, within the related views.py, or in the corresponding manager library. I quickly learned that code within a Django template doesn't seem to handle any parentheses or parameters, so that left views.py or the manager classes. I ended up working with both of these.

Groups are closely related to teams, so there was a fair amount of existing code to review and cannibalize. But, as with any complex system, it is hard to quickly build a mental map from the bottom-up like this. I felt a bit like I was redecorating while stumbling around in a dark room.

I got my Group Scoreboard together, though. I also built two Group Resource Use widgets. This proved a little trickier since it involved "subclassing" a widget. I also initially missed the INSTALLED_COMMON_WIDGET_APPS setting in settings.py, so I couldn't figure out why my common template file couldn't be found while resource_goal's could be. Also, I made a few errors that prevented the server from starting, and so I spent a while trying to debug single-line error messages such as "Error: cannot import name Team". I learned about this command:

  ./manage.py shell --traceback

which provides a full stack trace on an error. This was invaluable for debugging.

By this point, I was running short on time. I skipped making a Group Prize widget and started on a couple group-related Status page widgets. I spent an hour or so trying to decide how best to gather the group data from the team scoring/ranking system that is spread between score_mgr and team_mgr. I started toying with a plan to filter the team data to include only those in one group. As I neared the point to start testing it, I realized I didn't know exactly how the existing score widgets were placed on the Status page.

Specifically, I found the two templates (team_point.html and team_participation.html) that I wanted to copy in the widgets/status/score folder. However, simply adding a new group_point.html template in the same place didn't auto-magically show up. There is also no status.score widget registered through the Makahiki interface. Instead, these templates appear to be controlled indirectly through a separate scoreboard app. Looking through scoreboard's template was a bit of a mystery, however, since it makes no explicit reference back to the status.score templates.

The listed Makahiki widgets (front right) and corresponding file structure (back left).

At this point, it was time to move on from this project to bigger and better things. Here is a shot of my successful widgets currently on the Profile page:

And these are available within Makahiki to be added to any page:

Makahiki Configuration

Building on last week's Makahiki installation, this week I played around with some common configuration tasks. Below are the steps I completed. The section numbers given in parentheses correspond to the Makahiki docs found here.

0. Update your Heroku Makahiki instance (2.1.1.2.11)

I added the AWS credentials environment variables to the project's activate file in ~/.virtualenvs folder first. You should also add the MAKAHIKI_ADMIN_INFO here too. Uploading took 5 minutes or so.

I noticed this line in the output:

Environment variable MAKAHIKI_DATABASE_URL not defined. Exiting.

It didn't seem to cause any later problems though.

1. Getting to the challenge design page (2.3.2 / 2.2.1.1)

I couldn't get to the /account/login page at first. This took me 20 minutes to debug, and I reinitialized the Heroku instance during the process. It turned out I had just mistyped "account". <sigh>

2. Design the global settings (2.3.3.1)

I changed the Name and Logo settings as a test.

3. Design the teams (2.3.4.2)

I added a new Lehua-C team as part of the existing Lehua Group.

4. Set up users (2.3.4.3)

I could create new users, but I could not login normally using their accounts. It was possible to login as them from the admin's account, though. It turns out this was due to having mixed-case usernames, so make your usernames all lowercase.

5. Specify the games to appear in your challenge (2.3.6.1)

I disabled the "Drop Down" water game.

6. Learn about how to design the resource goal games (2.3.6.2.1)

This step just involves learning about the system and the design goals behind it.

6.1. Configure the Energy Goal Game for your new team

I switched the Lehua-C team to using manually-gathered energy data.

7. Learn about how to design Smart Grid Games (2.3.6.3)

This is another learning-about-the-system step, but it's important for the next few sub-steps.

7.0. Design on paper

Or at least mentally sketch out the next level your want to create in the Smart Grid Game.

7.1. Create a Level

I create a new Level 4 that unlocks once the user has completed a couple Level 2 tasks.

7.2. Create a new Activity action

I created an activity involving airship research.

7.3 Create a new Event action

I created an excursion to the helium fields.

7.4 Create a new Commitment action

I created a commitment to hold one's breath while passing a graveyard.

7.5 Finalize the grid

I edited the existing car-pool activity and added it to my level. (I was careful to select an activity that wasn't assigned to a category or level yet.)

Although I tested each of these sub-steps as admin along the way, I ran through the tests again with a different user.

8. Design the Top Score Game (2.3.6.4)

I created a new prize.

9. Design the Raffle Game (2.3.6.5)

I created a teddy bear raffle prize and tested it.

10. Design the Badge Game Mechanics (2.3.6.8)

I created a badge and earned it.

11. Manage Action submissions (2.4.2.2.1)

I approved and rejected a few activity responses that I generated during the earlier tests.

Overall, I found Makahiki to be a fairly polished and complete system. The most challenging parts of this proved to be my own mistakes (mistyping the URL) or undocumented requirements (such that the usernames need to be all lowercase). My general impression, though, is that Makahiki design and management is a rather long and tedious affair. I tried to create some slightly amusing activities to stay engaged with the process, but I'm still glad I'm done with this exercise.

Remote Makahiki Installation: A Reader's Guide

After installing Makahiki locally, it's time to try deploying it to Heroku. Once again, I'm following the user manual's instructions for this with a report of how each step went.

2.1.1.2.1. Install Heroku

Already installed.

2.1.1.2.2. Add your SSH keys to Heroku

Already done from an earlier project.

2.1.1.2.3. Verifying your Heroku account

Grudgingly done. I didn't really want to hand out my credit card for this.

2.1.1.2.4. Setup Amazon S3

Grudgingly done. (Later, in Step 9, I learned I should have selected US Standard as the region for my bucket.)

2.1.1.2.5. Setup environment variables

As instructed, though I changed the admin password.

2.1.1.2.6. Download the Makahiki source

Skipped this step, since I still have it from the local installation.

2.1.1.2.7. Initialize Makahiki

This was probably unnecessary, but: I wasn't sure if this initialization would mess up my existing local installation. I created a new virtualenv with mkvirtualenv makahiki-heroku.

In the next step, I realized I also needed to run pip install -r requirements.txt at this point.

The upload process took about 20 minutes or so.

2.1.1.2.8. Start the server

Everything loaded fine, but the pages lacked style and images. I checked my S3 bucket and it was empty. Trying again, I found this in the output:

Do you wish to continue (Y/n)? Y
resetting the db...
Resetting HEROKU_POSTGRESQL_RED_URL (DATABASE_URL)... done
syncing and migrating db...
Running `python makahiki/manage.py syncdb --noinput --migrate --verbosity 0` 
attached to terminal... up, run.4074
collecting static and media files...
Traceback (most recent call last):
  File "/home/ztomasze/makahiki/makahiki/manage.py", line 9, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
s3put -q -a AKIAJC76TME6N23PKRMA -s [...omitted...] -b ztomasze-makahiki 
 -g public-read -p `pwd` media
sh: 1: s3put: not found
s3put -q -a AKIAJC76TME6N23PKRMA -s [...omitted...] -b ztomasze-makahiki 
 -g public-read -p `pwd`/site_media site_media/static
sh: 1: s3put: not found
loading base data...

Both django.core and the s3put commands seem to be missing. I remembered that I had created a new virtualenv, so I ran pip install -r requirements.txt and tried again. Problem solved.

2.1.1.2.9. Verify that Makahiki is running

The links to images and stylesheets were still failing, even though the files were now in place on S3.

As an example, in the source code of the main page, the logo gif had this URL: https://s3.amazonaws.com/ztomasze-makahiki/static/images/old-logo.png

Accessing this URL directly got me this error message:

<Error>
<Code>PermanentRedirect</Code>
<Message>
The bucket you are attempting to access must be addressed using 
the specified endpoint.  Please send all future requests to this 
endpoint.
</Message>
[...snip...]
<Endpoint>ztomasze-makahiki.s3.amazonaws.com</Endpoint>
</Error>

And indeed, accessing https://ztomasze-makahiki.s3.amazonaws.com/static/images/old-logo.png instead worked fine.

Feedback from Yongwen, the current Makahiki admin, suggested that this URL difference is due to which region you select when creating the S3 bucket. I originally went with S3's suggestion of Oregon, but apparently US Standard gives you the URLs assumed by Makahiki. To try this, I deleted and tried to recreate my bucket. There was a temporary name conflict, so I had to choose a different bucket name. I updated the MAKAHIKI_AWS_STORAGE_BUCKET_NAME environment variable and reinitialized Makahiki (step 7).

It looks like everything now works.

Conclusion: Total Time: 3 hours. This was longer than the local install, although there were fewer steps. This time difference was mainly due to waiting for downloads and uploads. Overall, this process was about as painful as any Heroku deployment, though each one gets easier with more practice.

Local Makahiki Installation: A Reader's Guide

Today I worked through the installation of the Makahiki energy competition platform, as documented here. This is a record of how each step went for me. YMMV.

2.1.1.1.1.1. Hardware requirements

Running on Debian 7 in a VirtualBox with 2GB of RAM (on Windows 7), as described earlier. Apparently I could have just grabbed a large VirtualBox image and worked from there. Instead, I followed the step-by-step Unix instructions within my existing system.

2.1.1.1.1.2. Install Python

I checked the version: already installed.

2.1.1.1.1.3. Install C Compiler

I checked the version: already installed.

2.1.1.1.1.4. Install Git

I checked the version: already installed.

2.1.1.1.1.5. Install Pip

I checked the version: already installed.

2.1.1.1.1.6. Install Virtual Environment Wrapper

Followed instructions. As root: pip install virtualenvwrapper

I initially moved on at this point, but I later found that workon did not work. Don't forget to setup up your shell startup file as instructed. I chose to go with lazy initialization, and I changed PROJECT_HOME to just $HOME, since that's where I've been putting my projects so far.

2.1.1.1.1.7. Install Python Imaging Library

As instructed, I used apt-get to install python-imaging and libjpeg-dev. (I already had python-dev.) I also had to set up the symlinks as instructed.

2.1.1.1.1.8. Install PostgreSQL

PostgreSQL was already installed, but I did need to edit pg_hba.conf as instructed. The full path to this file was: /etc/postgresql/9.1/main/pg_hba.conf

2.1.1.1.1.9. Install Memcache

As instructed, I used apt-get to install memcached and libmemcached-dev.

2.1.1.1.1.10. Download the Makahiki source

As instructed.

2.1.1.1.1.11. Workon makahiki

Once I fixed the part I accidentally skipped in step 6, this worked fine.

2.1.1.1.1.12. Install required packages

As instructed. The download and install took a while (5 minutes) with a lot of output and warnings, but no errors.

2.1.1.1.1.13. Setup environment variables

As instructed. I added the configuration to the end of ~/.virtualenvs/makahiki/bin/postactivate. If you copy-and-paste, don't forget to remove the % signs. I also reused my existing PostgreSQL user, django. (This may or may not be a good idea, so I don't necessarily recommend it. It worked fine for me, though.) Once I wrote these to the config file, I typed deactive and then workon makahiki to refresh the settings.

2.1.1.1.1.14. Initialize Makahiki

As instructed.

2.1.1.1.1.15. Start the server

As instructed. Both servers worked for me.

2.1.1.1.1.16. Verify that Makahiki is running

Yay, it runs!

Conclusion: This process is fairly easy as long as you go carefully and don't miss a step. It is less difficult than installing the Django toolchain for Heroku, though you would need to make it through much of that process first if you want to deploy Makahiki to Heroku.

Total Time: 2.5 hours, including the time taken to document the process here. Probably only 1.5 hours or so if you were just following the directions.

Kukui Cup: A Taste of Gamification

I spent this past week playing a demo run of the Kukui Cup competition. This is an online educational "game" normally run for UH undergrads living in the dorms. It intends to teach participants about current energy issues and promote new energy conservation habits.

The primary form of interaction in the Kukui Cup is completing "Get Nutz" tasks. These tasks include watching educational videos and answering a short question about each one; commiting to a daily habit such as turning off lights when you leave a room; and field-trip activites. Completing each task earns you points that you can apply to a raffle to win prizes. In the real competition, real-time energy use of the dorms would also play a role.

It was interesting to see some recent views on gamification embodied in the mechanics of the Kukui Cup. Jane McGonigal proposes that gamification could be used to harness the energy people devote to games to affect positive change in the real-world. The Kukui Cup seems founded on that principle: that the habits and knowledge gained during the course of the game will continue to influence people afterward.

Tadhg Kelly suggests that gamification really boils down to three mechanics: social validation, completion, and prizes. The Kukui Cup uses all of these. You can see your ranking compared to others, and you are encouraged to advertise your participation through social media and on your dorm-room door. The Get Nutz layout makes it very clear what tasks are available to you and how far you've progressed through them, which gives a sense of completion. The points you earn are all given a real-world value by letting you use them to buy tickets for raffle prizes.

It is less clear to me whether the Kukui Cup contains the three ingredients that Sebastian Deterding claims are missing from most gamification efforts: meaning, mastery, and autonomy. When you strip away the game mechanics of the Kukui Cup, there is not much content left behind that is inherently meaningful to users. That is, the game needs to assume that user have an existing motivation to learn about energy and compete with their peers to conserve it; the game then provides a fun way to do so. There is no real skill that you improve by playing the Kukui Cup, although the way that Get Nutz is broken down into levels that need to be unlocked does give a sense of improvement over time. Finally, the Kukui Cup offers users a fair amount of autonomy in choosing which tasks to complete and habits to cultivate, but users are not able to propose new activities or incorporate goals of their own design into the game.

I also gathered a few practical design tips from playing the Kukui Cup. I found the steady stream of points to be almost motivation in themselves, though giving them a real value through the raffle system was also fun. It was nice to be in the running to win something even if I wasn't close to the top player's ranking. The task-unlocking system was also motivating; it felt a bit like leveling-up in an RPG. The need to check back in every day in order to claim the points for yesterday's habits was a very slick way to keep people logging in. That said, it was fairly easy to cheat on the habits, and sometimes it was unclear whether I adhered to the rule or not. For example, if I promised to do only full loads of laundry, but I didn't do laundry at all today, did I still fulfill my promise?

On the downside, the constant pressure to advertise what I had done on Facebook was a bit annoying, as were the constant requests for feedback. Because of these, it took 3 or 4 clicks to complete a task when it seemed like it should have only taken 2 or 3. Also, if the review questions for the short videos were multiple-choice or matching rather than short answer, their scoring could be automated. This would eliminate the delay in earning points and save someone a few hours "grading" time.

Overall, I found the Kukui Cup a solid example of gamification. It's always nice to explore some concrete examples of theoretical concepts.

Django on Heroku: Revisited

Last week, I spent quite a while getting a simple "Hello World" Django app in place on Heroku. This week, I wanted to see if I could repeat the process by posting a small working demo project. Things went much more smoothly this time around.

I used git to grab a local clone of the project. I used virtualenv to setup a virtual environment and pip to load that environment with the project's requirements.

I created a local PostgreSQL database so I could test things locally. I remembered to set the DATABASE_URL environmental variable and to sync Django with the new database. I debated on whether I needed to run python manage.py collectstatic in order to move the website's static files (such as scripts and stylesheets), but the site ran fine locally as-is without it.

Since everything seemed to work fine locally, it was time to push to Heroku. I double-checked that .gitignore was in place with venv and *.pyc in it. When I ran git add . I realized that I'd done all of this testing without touching a single project file. That's a neat design.

I created a new heroku app and used git to push the files to heroku. I initially forgot to sync the database on Heroku's end, but the resulting error message was obvious and it only took a minute to correct. To make sure everything really worked, I used Django's default admin interface to add some database contents (a poll with 4 possible answers). Everything seems to be running fine.

Given the light expected traffic, I didn't bother to switch out the default Django server for gunicorn.

All of this took me about 1 hour, compared to last week's 20 hours. Now that I've got this down, maybe Heroku isn't so bad after all!

Django on Heroku: A Reader's Companion

I recently setup a fresh VirtualBox Debian Linux system. Within that context, I then walked through the Heroku tutorial on Getting Started with Django on Heroku. I ran into a number of errors along the way. Most of them took a few minutes (or an hour) of surfing to find a solution. If you've found this page, perhaps you hit one of those same error messages, and so my solutions will save you a little time.

I'll follow the sections of the current Django on Heroku tutorial here and just add the errors and solutions I encountered.

Prerequisites

There are a bunch of different Python installers out there: pip, distribute, setuptools, easy_install, etc. You'll need both pip and virtualenv... but virtualenv is listed specifically as a prereq at the start of the tutorial. According to the virtualenv page, the preferred way to get virtualenv is to use pip... but the best way to get pip is to use virtualenv. To resolve this, I decided to install pip globally... but pip requires distribute. Of course, the easiest way to get distribute is to use pip to install it (haha!)... but there is also this option where you use curl (or wget) to download an installation script. Once you have distribute, you can do something similar for pip.

Start a Django app inside a Virtualenv

It turns out the virtual environment needs a few things installed locally too...

Error: pg_config executable not found.

You need the postgresql package installed. (Use apt-get install postgresql to install it.)

Error: You need to install postgresql-server-dev-X.Y for building a server-side
extension or libpq-dev for building a client-side application.

Okay, so you also need the postgresql-server-dev-9.1 package (currently).

In file included from psycopg/psycopgmodule.c:27:0:
./psycopg/psycopg.h:30:20: fatal error: Python.h: No such file or directory
compilation terminated.

And the python-dev package too.

Hooray! At this point, I could get a Python app created in the virtualenv and run it. Things seemed downhill for me from here until...

Deploy to Heroku

Typing heroku create gave me:

!    Heroku client internal error.
getaddrinfo: Name or service not known (SocketError)

This error was coming from the excon gem. heroku login gave me the same error. Turns out I had a DNS problem. I entered 8.8.8.8 (Google's public DNS server) into /etc/resolve.conf and tried it again. Worked.

Then git push heroku master lead to:

Permission denied (publickey).
fatal: The remote end hung up unexpectedly

I needed to generate a public key with: ssh-keygen -t rsa By default, this puts the key into ~/.ssh/id_rsa.pub. You then need to register it with Heroku: heroku keys:add ~/.ssh/id_rsa.pub

At this point, I could execute the basic Django app both locally (with foreman start) and on Heroku.

Although my app wasn't actually using the database, I figured I'd better try syncing with it anyway...

Syncing the database

I wanted to run it locally first, so I needed to setup a local postgreSQL database. (The tips on the assignment page were very helpful here.) Also, you need to be postgre user in order to execute the psql command that starts the PostgreSQL console. (It took me a while to even track that command down.) So:

root@tinkVr:~# su postgres
postgres@tinkVr:/root$ psql
could not change directory to "/root"
psql (9.1.8)
Type "help" for help.

postgres=# CREATE USER django WITH CREATEDB PASSWORD 'djangoapp';
CREATE ROLE
postgres=# CREATE DATABASE a17 OWNER django;
CREATE DATABASE
postgres=# \q

You'll probably want to change the lowercase details (user, password, and database names) to something else. Note that you do not need to update your Django project's settings.py with these details because dj_database_url will pull these from from the DATABASE_URL environment variable. (If you log into Heroku's web interface and check your database settings, you can see what these values will actually be in the Heroku environment.) However, to run locally, you do need to set this variable appropriately. In my case, this translates to:

export DATABASE_URL=postgres://django:djangoapp@localhost/a17

typed at the command prompt.

At this point, I could successfully run python manage.py syncdb locally. However, heroku run python manage.py syncdb lead to a timeout. This happened even when just trying to open a simple shell on Heroku with heroku run bash. Heroku does offer some troubeshooting on this. It turns out port 5000 was being blocked by my ISP (the wired network in the Computer Science department on campus). I switched to the public UH wireless and I was able to connect.

The rest of tutorial went smoothly for me. (pip installs gunicorn for you.) Hopefully it goes smoothly for you too!

Next up: Fleshing out the Django app to actually do something...

Debian 7 on a VirtualBox

I'm currently taking a class on serious games. We've already covered Django and the nature of gamification. This week we are trying to deploy a simple Django project to Heroku, a cloud-based app hosting service. This proved fairly difficult, requiring that we get a long toolchain in place. (Details of that process to follow.)

This toolchain requires running scripts that download and install other required components. Whenever you're missing something, you get error messages that look like this:

  building 'psycopg2._psycopg' extension
  error: Unable to find vcvarsall.bat

You then have to figure out why you don't have vcvarsall.bat, or why it's not in the right place, and what you need to do to fix it.

Other students in the class reported that, since most of the required tools assume a unix-like environment, trying to get everything configured in Windows seemed to lead to even more problems. Also, project dependency managers like Ivy or pip always make me a little nervous. I don't really like the idea of downloading some dude's script from GitHub, running it, and having it download other arbitrary code from various points around the web and freely executing it all on my machine.

Given this context, I decided to create a Linux environment to work in. While I've dual-booted on a number of different machines before, I decided to try a virtual machine this time. There are a lot of advantages to this:

• No need to mess with your current partition configuration.
• Your can run the virtualized OS in a window, switching back and forth between the environments, even copying clipboard and file data between them.
• You can pause and save a running instance. (I've already used this feature: I paused a lengthy compilation/installation process so I could shutdown the real machine and then resumed everything again in the morning.)
• You can take a snapshot of the whole machine as a backup, which can be especially handy before you make any significant OS changes.
• You can copy the entire virtual OS environment to another machine. So, once you get your OS all configured, you're not bound to the original hardware.

For my virtual machine, I went with VirtualBox over VMware, since VirtualBox costs nothing and is open-source. Their provided documentation is pretty good too.

I then installed Debian 7 on it. (I actually did this twice, but here's what the process looked like the second time around. I'm assuming here that you know the basics of installing a Linux system.)

Due to previous Linux endeavors, I already had an ISO of the last stable Debian release (6.0/"squeeze") handy on a CD. (If I was doing this from scratch, I would grab a "testing" ISO instead.) I created a VirtualBox machine with a 12.0GB hard drive. (I hear this the only parameter you can't change once you create the machine.) Although it's a little slower, I went with a dynamically-sized file to save space on my host machine. I gave the machine 2GB of RAM and 1 CPU. (I'm used to running Linux on ancient hardware, so this is more than most of my installations get.) When I first started the virtual machine, it booted off the CD and began the installation wizard.

Debian's default Graphical Install worked out fine. When asked for what subsystems to install, I unchecked everything (including Standard System Utilities), so I had the barest machine possible. The default setup reserved 500MB of my 12GB for a swap partition.

The "stable" release tends to get outdated. For example, squeeze gives you Python 2.6. The current 2.7.3 is available in the "testing" release, and that will be updated within about a week of any new Python updates. So, once the installation was done, I edited my /etc/apt/sources.list to pull from "testing" rather than "squeeze". I then ran an apt-get dist-upgrade to switch over.

In upgrading, I got a new kernel. On rebooting, I found that my virtualbox guest services were not longer starting up properly. I would get this error:

AdditionsVBoxService error: VbglR3Init failed with rc=VERR_FILE_NOT_FOUND

At this point, the only obvious problem this caused is my mouse scrollwheel stopped working, but I knew it could cause other problems later on. I eventually learned that I needed to install the appropriate linux-header package. (While I love the idea of Linux, this is an example of why I don't actually use it everyday. You upgrade something that's supposed to be routine and suddenly something else breaks with some arcane error that requires an hour to debug, some config file hacking, and some new packages to track down. <sarcasm>Oh, you're obviously just missing the necessary the kernel headers. Duh!</sarcasm>)

For my desktop environment, I used: apt-get install lxde. This is pretty minimalist desktop, but I figured it should still be snappy in the virtual machine. So far, it is responsive. It looks crude coming to it from Windows 7, but I find I don't mind so much when I have Windows 7 running in the background, a single click away! When I get some time, I'm curious to see how well the latest KDE runs under these conditions.

I also installed openjdk7 and Eclipse (which wanted openjdk6 and a bunch of other stuff).

At this point, I had a working virtual Linux machine. Time to put it to work!

LXDE in maximized window with Windows taskbar readily available below.

Global Game Jam 2013: The Queen's Heart

I had fun hacking together a game called The Queen's Heart over the weekend as part of the Global Game Jam 2013. More info about the result can be found on its new project page.

In other unrelated news, I worked on picking up Clojure over the break, though that's lagged a bit now that another semester has started. I'm currently learning Git and Django (the Python-based web framework) as part of a class on serious games. All fun stuff!

Keyboard and Monitor Efficiency-Boosts

In online discussions, I've often seen two claims made about efficiency:

• Two monitors are better than one
• Master gurus work so fast because they never take fingers off the keyboard.

Over the past month or so, I've been exploring these for myself.

First, I bought an affordable 19" ViewSonic monitor for about $100 from Amazon. I also picked up a desk clamp, and so I was able to install it directly above my 15" laptop screen. The two are the same resolution, and, if I slide the monitor back back 3 or 4 inches, they're about the same size in my visual field. So I now have double the screen real-estate. I also installed a copy of UltraMon.

So far, the second monitor has been fun, but not all that useful. It's handy when I want documentation or my streaming radio up at the same time I'm working on some code. However, I find that two screenfulls of documents with white backgrounds are pretty bright at night, even though I cranked the brightness down pretty low. Also, even though the monitor is straight ahead of me, it makes my neck a little stiff if I work on it for too long. I guess I've gotten used to that slightly downward sightline of a laptop screen. Still, I don't regret the purchase and I'm fine with flipping the second monitor on whenever it's useful. I think (or at least hope) I'll gradually put it to more use over time.

The second notion I've been exploring is that keyboard-based computing is faster than mouse-based. This is often an argument for using emacs or (even more so) for vi. As mentioned previously, I've been playing with AutoHotKey. Here are a few of the interesting settings from my AutoHotKey script file:

#KeyHistory 0   ; don't record keypresses

I added this when I realized all my keystrokes--including the title of the current window--were being logged. Don't really need that info lying around.

~RAlt::Shift
~RCtrl::Shift

These turn right Alt and right Control into Shift+Alt and Shift+Ctrl. I've found this very handy with various macro and other keybindings.

This next setup was quite interesting:

CoordMode ToolTip, Screen   ;(at top of file)

SetCapsLockState AlwaysOff  ;;CAPS button does nothing alone

CapsLock & Enter::
 if GetKeyState("ScrollLock", "T") {
   ToolTip  ;turn it off
 }else {
   ToolTip NavMode, 0, 0
 }
 Send {ScrollLock}
 return
 
#If GetKeyState("ScrollLock", "T")
  i::Up
  j::Left
  k::Down
  ,::Down
  l::Right
  u::Send {PgUp}
  Space::Send {PgDn}
  h::Send {Home}
  `;::Send {End}
  a::Send {Alt Down}
  a UP::Send {Alt Up}
  s::Send {Shift Down}
  s UP::Send {Shift Up}
  d::Send {Ctrl Down}
  d UP::Send {Ctrl Up}
  f:: ;reserved for... mouse controls?
#If

CapsLock & i::Send {Up}
CapsLock & j::Send {Left}
CapsLock & k::Send {Down}
CapsLock & ,::Send {Down}
CapsLock & l::Send {Right}
CapsLock & u::Send {PgUp}
CapsLock & Space::Send {PgDn}
CapsLock & h::Send {Home}
CapsLock & `;::Send {End}

The idea I had here was that the different keyboard modes (ScrollLock, NumLock, CapsLock, and InsertMode) were going largely unused, but each could allow the keyboard to be completely remapped. My laptop keyboard doesn't even have a ScrollLock or NumLock key, and lacks any keyboard indicator lights. I thought I'd start with the least used--ScrollLock. Hitting CapsLock+Enter (a double-pinky-tap) puts me into ScrollLock and, since I don't have an indicator light, pops a very small "NavMode" tooltip in the upper left-corner of the screen. The homekeys (jkli,) become arrow keys, with h and ; as Home and End and U and Space as PageUp and PageDown. a, s, and d became alt, shift, and ctrl, respectively. You could of course take this further.

The homekeys-to-arrow-keys conversion would be handy enough to use without switching modes, so the second part means I can hold down CapsLock to turn the homekeys into arrow keys.

So, there we go: no reason to ever leave the homekeys again!

I found I hardly ever used this. I'm already trained on the arrow keys and the mouse, so that's where I go. If I'm moving my cursor far, I grab the mouse. If I'm going up a line or two, just a couple arrow-key taps are good enough. Sliding my right hand over 3 inches to use either of these wasn't costing me that much time, but trying to remember to use CapsLock instead was.

I still think it's a cool idea, though, which is why I'm sharing it here. Also, an approach like this would be very handy to implement a working number pad on my laptop, since it doesn't provide one.

In the end, I reverted to:

+CapsLock::CapsLock  ;Shift+CapsLock toggles CapsLock mode
CapsLock::Ctrl

which just turns CapsLock into Control.

I found I wasn't using this combo much a month ago, but this past couple days I've been exploring the fancier features of Eclipse. Ctrl+Space in particular is much handier with CapsLock as Ctrl.

Overall, my foray into keyboard efficiency has been disappointing. Supposedly the learning curve is worth it, but I doubt the extra few milliseconds are ever going to add up to the time it will take me to remap and retrain to do it. This old article suggests that my feelings on this are perhaps not so far off after all: that the speed of keyboarding is actually illusory.

This got me thinking of the QWERTY vs DVORAK debate. Supposedly, for experts, DVORAK is 10% faster (though this claim is contested). But, even if it is, I'm not hitting the QWERTY limit. That is, other people can type way faster on QWERTY than I can. I don't think it's primarily the keyboard layout that's slowing me down there; it my physical and mental processes that are too slow.

Bringing this back to the keyboard-vs-mouse debate, I see the same thing at play. I hardly ever mouse through a menu during my normal work, which I think is the main source of the "GUIs are slow" argument. I do use common keyboard shortcuts (like Ctrl+Z/X/C/V), etc, which are very fast when combined with mouse selection. When I don't use a keyboard combo, I generally hit a toolbar button. This is about as fast as a mouse-gesture. And even heading into a well-known and shallow menu doesn't really take that long. Striving for micro-improvments in this area is not really worth it to me. (Your mileage may vary, though, which is fine.)

This brings me in turn to my recent Eclipse experience. I'm realizing that Eclipse--with its Ctrl+Space and Alt+/ completions and various Content Assists and Code Templates--can drastically improve code generation times because it writes large sections of the code for you. I suspect that this is what may really be at work with older editors like emacs and vi: not that they are keyboard-only but that they are so full of useful tools. It's the intelligent and useful tools that make you faster, not the milliseconds you saved in selecting the tool.

So I've decide to give up on this idea of converting to entirely keyboard use. Instead, I'll be embracing my mouse and toolbar use. (For example, I mapped WinKey+ScrollUp to switch screens between monitors.) I'll see what I can do about speeding those up. I'll still use the keyboard and good bindings where appropriate, of course, but writing good macros is going to save me way more time than not moving my hand 3 inches over to the arrow keys.

I have one last insight on this, which is two-fold. First, keyboard sequences (such as Alt+F, S for Save on Windows or Ctrl+X, Ctrl+S on emacs) are about as fast as a more complicated finger-spreading key-combo (such as Ctrl+Alt+S). Secondly, menus actually provide and scaffold such combos. Alt+F gets you the File menu, and then every element there has a corresponding letter. You can get to any menu option in 2 or 3 keypresses--just like an Emacs key-combo. The difference is that the menu's always there to remind you of your options, and the menu path actually informs the keybinding. For example, in Eclipse, Ctrl+F11 is Run. I find this hard to remember. Why F11 and not some other function key? But consider: Alt+R, R, Enter. Not much longer, more compact on the keyboard, and memorable because it corresponds to: Run Menu, Run, (Execute). (The Enter is required because there is more than one R-bound menu option in the Run menu.) Sequences like this also give you more keybinding options at a factorial rate. For example, if you make Alt+J the start of all your Java-based macros, you get 26 three-key bindings (or many more, given all the other non-letter keys) with that short and relevant intro mnemonic.

Anyway, I'm sure I'll continue to think on this, but I'll be putting more effort now into intelligent macro design and Eclipse tool use than trying to speed-up my keyboard use. So, if you're still a coder putting your mouse to good use, you're not alone!

A quick fix to improve Python3 startup time

My web server is very low-end. Dating from the mid-90s, it has a 200Mhz Pentium and 96MB of RAM. It was running Debian 2.2 (potato), but I recently upgraded to the most recent Debian 6.0 (squeeze). I'm impressed that it even runs.

I also upgraded to Python3 in order to handle the recent overhaul of my Tamarin automated grading system. I compiled Python3 from source to do this, since Debian doesn't include it yet. They're still shipping Python 2.6.

Tamarin is little more than a bunch of CGI scripts. I expect it to run a little slowly on this machine, but, after the upgrade, any CGI request is taking about 4 seconds, which is fairly intolerable. Static webpages are still responsive enough, though. So I started up the Python3 interpreter... and waited. Yep, that's where the lag is. See for yourself:

ztomasze@tamarin:~$ time python3 -c 'pass'

real    0m2.493s
user    0m2.336s
sys     0m0.160s

Here, I'm just starting python3 to execute a single 'pass' statement that does nothing. This takes 2.5 seconds.

I read somewhere that not loading the local site libraries by using the -S option can give a performance boost. Since Tamarin uses only standard modules, I gave it a shot:

ztomasze@tamarin:~$ time python3 -S -c 'pass'

real    0m0.465s
user    0m0.392s
sys     0m0.072s

A 500% improvement! I even installed the default Python 2.6, just to compare times:

ztomasze@tamarin:~$ time python -c 'pass'

real    0m0.448s
user    0m0.348s
sys     0m0.060s
ztomasze@tamarin:~$ time python -S -c 'pass'

real    0m0.185s
user    0m0.148s
sys     0m0.036s

So Python3 is significantly slower for me than Python 2 was, but using the -S option at least gets me back to standard Python2 times.

This savings didn't really translate directly to improved CGI preformance though. Running two of my scripts from the command command line, I experienced the following:

                       status.py upload.py
original time          4 sec     6 sec
adding -S to #! line   3 sec     5 sec

A delay this long is still fairly intolerable. And I don't think the lag is inherent to Tamarin, since the delays weren't this long with Python2 and Debian 2.2 on the same machine.

I know I could probably shave off some more time for Tamarin by using FastCGI, SCGI, or mod_python. SCGI looks most useful to me given my existing codebase. Whenever I get some free time, I'll look into that.

Efficiency though Keyboard Shortcuts

One of the things I learned from the discussion of my recent Lisp posting was the value of delegating work to a good editor and the possible speed gain of good keyboard shortcuts. While I did not agree that the best path to these goals was necessarily through Emacs, I did decide to try to use my keyboard a bit more efficiently than I have been.

First, I reviewed the various keybindings already used by my OS, Windows 7. Hey, that Windows Logo key actually does have a few valuable uses! I also realized that I never use my function keys very much.

I also installed AutoHotKey and tried a few simple useful bindings, including:

• Capslock is now Ctrl. (Shift+Capslock acts as normal Capslock)
• Right Alt and Right Ctrl are now equal to Shift+Alt and Shift+Ctrl.
• Win+q quits a program (like Alt+F4) and Win+c opens a command prompt.
• The right-click menu button (AppsKey) is now a special function key. For example, I use it with various letters as shortcuts to certain directories when I'm in Windows Explorer.

I'm still working on actually using some of these on a regular basis, though. (Old habits die hard.) I'm also trying to use my alt keys with my thumbs without taking my fingers off the home keys, and using tab (replaced appropriately with spaces) more often when coding.

Anyway, it's all rather nerdy, but the possibilities are exciting. If you work on Windows, check out AutoHotKey. You may want to customize your system across all your different applications--especially if you start thinking about all the hundreds of possible key combos currently going unused on your keyboard!

Unit-testing a Python CGI script

This summer I'm overhauling Tamarin, my automated grading system. Under the hood, Tamarin is little more than a bunch of Python CGI scripts. However, as I overhaul it and convert it from Python 2 to 3, I also wanted to build a proper unit test framework for it.

It's been a dozen years or so since I last used Perl and CGI.pm, but I recall running my scripts on the command line and manually specifying key=value pairs. So, I was somewhat surprised to find no comparable way to test my CGI scripts in Python. The official Python cgi module documentation suggests the only way to test a CGI script is in a web server envirnoment. That's an unnecessarily complex environment for quick tests during development and precludes any simple separate unit tests.

In general, I'm not very impressed with the cgi module docs. In fact, browsing around revealed that there are a number of parameter options undocumented in the official docs.

Using this found information, I was able to build my own cgifactory module. Depending on the function called, it allows you to build a cgi object based on either a GET or POST query. For example:

  form = cgifactory.get(key1='value1', key2='v2')

If you then write your CGI script's main function to take an optional CGI object, you can easily build a CGI query, pass it to your script, and then run string matching on the (redirected) output produced by your script. Of course, most of your unit tests will probably be of component functions used by your script, but sometimes you want to test or run your script as a whole unit. cgifactory will help you there.

The cgifactory code is available here, where you'll always find the most recent version. The code itself is actually quite short; most of the file is documentation and doctests showing how to use it. I don't guarantee it's right, but it's worked for me so far. Hopefully it might be of use to someone else too! Feel free to copy, modify, and/or redistribute.

(Oh, and if you really need a command line version, it shouldn't be too hard to write a main that parse key=values pairs into a dictionary and then calls cgifactory.get(pairs) to build the CGI object.)

Bug: C enum variables stored as unsigned ints

I read in K&R that enum values are basically int constants (like #defines in that way), and so enum variables are equivalent to ints. However, in C (not C++ though), you may assign any int value to an enum variable--even if that int value is not one of the listed values in the enum definition. You can do this without even raising a compiler warning.

In a program I was working on, I took advantage of that. I had an enum of values 0 through 7:

  enum direction { N, NE, E, SE, S, SW, W, NW};

In a particular function, I was scanning a map for a target in different directions and decided to return -1 if there was nothing interesting found in any direction. However, this led to strange bug.

The following program shows this bug clearly:

#include <stdio.h>

enum nums {zero, one, two, three};

int main(void) {  
  //using an enum as normal
  enum nums myNum = zero;
  printf("zero == %d\n", myNum);
  
  //assigning int value to an emum
  myNum = -1;
  printf("-1 == %d\n", myNum);
  if (myNum >= 0) {
    printf("%d >= 0\n", myNum);
  }else {
    printf("%d < 0\n", myNum); 
  }
}

This program prints:

 zero == 0
 -1 == -1
 -1 >= 0

I'm using GCC, and the manual itself says: "By default, these values are of type signed int" and "Although such variables are considered to be of an enumeration type, you can assign them any value that you could assign to an int variable".

However, further research shows that gcc will store an enum variable as an unsigned int if you have no negative values in your defined enum. For example, if I add neg = -1 as an extra value to my enum nums above, the output of the program changes to what I expect: -1 < 0.

Apparently the section 6.7.2.2 of the C99 standard (draft version) clarifies that this is allowed--that the particular int format used is implementation-dependent. An official version of the C90 standard is not freely available for comparison. -std=c90 doesn't change gcc's behavior on this issue.

Lisp: First Impressions

I learned the basics of Lisp 10 years ago when I was the TA for ICS313: Programming Language Theory. This semester, I returned to Lisp after a decade away, again in the context of TAing for ICS313. After so long away, I basically had to start over from scratch again. Though it certainly goes much faster the second time around, it's still a lot like learning it the first time. So these are my "first" impressions.

We are learning Common Lisp. So far, I'm not very impressed.

First of all, I find that Common Lisp is bloated. There are usually about 6 different ways to do something. For example, want to compare two items? Consider =, eq, eql, equal, equalp... and that's just getting started. Want to print something? Consider print, princ, prin1, pprint, format, etc. The reason that so many options exist is because each is subtly different. I'm sure that, once mastered, all these options increase your programming power, since you can pick exactly the right tool for the job. And, in many cases, some of the options seem to be now-unused holdovers that persist from earlier days in Lisp's development. So eventually you learn which ones you can ignore. But it all makes Common Lisp rather tedious to learn.

My next stumbling block is the formatting. As evidenced by both my students' code and my own inclinations, it seems natural as a programmer to try to format the parentheses to make matching them up much easier. The parentheses are subtle, and it's easy to get one out of place. This can produce code that compiles or loads but then fails at runtime because the meaning is subtly different to what you intended. This well-discussed blog posting sums it up fairly well. In general, I agree with the author there. Parentheses in Lisp mark the start and end of various expressions in the same way that braces mark the start and end of blocks in C-like languages. In C-like languages, we have proponents of such extremes as the Allman style of indenting. These Allman proponents feel that such formatting is so essential to readability that every brace deserves its own line! Yet the Lisp community advocates the exact opposite: that no parenthesis should be clearly placed. Instead, they should all just be tucked away at the start or end of a line of code. Supposedly some day you get to the point were you can "see past the parentheses". But this seems to me like a convention that makes code unnecessarily hard to read.

I find most of the arguments for this "parenthesis stacking" format are weak at best. One of them that irks me is that your editor will help you. First of all, you're not always reading code in an editor. Secondly, I should have to move my cursor around or press keys or require fancy color highlighting to make quick sense of the code on the screen. It's called "readability", not "navigability". A third argument is that you can ignore the parentheses because the indenting should show you the structure. But the indenting is not what actually determines the code structure--the parenthesis do! So I need to be able to quickly spot when the parentheses are wrong even though the indenting is correct.

This formatting thing bugs me because it seems the problem comes from an asinine coding convention choice. And that's the one reason that has me formatting this way rather than the way that makes sense to me: because "that's how it is done in Lisp." Like the choice to drive on the right side of the road, it's hard to buck the community on such a choice as a late-comer! Yet, because it makes the code harder to work with, but with no good reasons that I can see, it feels a little like hazing: "This is the way we all had to deal with it when we learned, so you do too."

That brings me to the Lisp community in general. First of all, I don't care to worship at the altar of Emacs. As I mentioned above, I shouldn't have to have a special editor to write code. Don't get me wrong: an IDE is great at increasing productivity and I definitely want one. But the code should be both readable and writable without one. Then I'm free to choose the IDE that meets my own requirements. (On that note, after trying Emacs and LispWorks, I settled on Cusp. It's a little tricky to get up and running, and a bit quirky at first, but it works pretty well. The highlighting of parentheses structure is very helpful.)

Secondly, there's just this "Lisp is so much better!" vibe in the community. Now, obviously you've got to wave the flag for your favorite language. I have no problems there. But, as others have pointed out, if Lisp is so wonderful, how come we're not all using it? Just about every computer science major has to learn a Lisp dialect at some point, so it's not just an issue of exposure. Is it because it lacks good libraries for modern tasks? Is it because, while powerfully writable, its dynamic re-writability makes it hard for someone else to read or maintain? Is it because, while a pioneer of so many cool ideas, most of those ideas have now been imported from Lisp by other languages? Is it because Lisp has built up over 50 years of cruft, but each new Lisp project to simply and overhaul it fractures the small but fanatical Lisp community, leading to inter-dialect derision and flamewars? Hard to tell. What I can tell is that the fanatical belief in Lisp's supremacy over all other programming languages is a little hard to swallow.

My general conclusion is that Lisp is still worth learning for the history of it. However, I don't think I'll be taking it up as my day-to-day language. I'll be looking elsewhere for more modern implementations of Lisp's contributions that are useful to me.

Still, the code-as-data idea is largely still unique to Lisp. That would be fun to explore more, so finding a Lisp dialect that fits me better might still be rewarding. I've considered Scheme a bit, but I think either Clojure or newLisp would be even better. (I found newLisp because I thought: "Why doesn't someone clean up and simplify Lisp back to it's glorious essence? If I did that, I'd call it 'new Lisp'." I searched... and behold! Already done 20 years ago by someone else.) Both seem to have newer, friendlier, more open communities. Clojure has the advantage of the JVM and the entire Java API behind it. newLisp is targeting what I think is a great niche for Lisp: scripting. This is where powerful writability at the expense of readability and maintainability is a viable tradeoff.

These are my current impressions of Lisp. Perhaps they'll change with time. If so, I'll let you know!

Group Programming: Some Adjustment Required

After last week's technical review, Jeff and I swapped projects with the grads group. That is, we took over on the grads' code, and they started working on ours. Both groups added three more commands to the existing codebase that they had just inherited.

This was an interesting experience taking over on an existing software project. Of course, there was some initial overhead getting familiar with the layout of the new project's code. This wasn't too onerous, though, since the overall structure was fairly logical.

Once Jeff and I started working, the work went smoothly. We had already completed one project together. I think Jeff and I communicated well, and each of us made some significant contributions. We agreed on all of the major design issues.

Still, for all of the harmony, I'd say that my most interesting learning experience this week was the old "pick your battles" experience. If you've ever shared a living space with someone as an adult--whether a college roommate or a romantic partner--you probably know what I mean. There's a period of adjustment to the person's habits and quirks. For example, maybe your new roommate already put the silverware in the left drawer when you feel that it is obviously more natural to have it in the right drawer. Or maybe your new lover seems to somehow constantly drop loose change; eventually stray coins lie scattered throughout the house.

For each new quirk you discover, you have to decide whether it bothers you or not. If it does, then you need to figure out if you're willing to just overlook it, constantly clean up after them, or make an issue of it. If you make an issue of every little thing, you can easily become an unpleasant nag. Depending on how it's done, cleaning up behind someone can come across as a passive-aggressive show of disapproval. But, if something really does bother you and you don't speak up, you can find that your living space is not really your own. That's what I mean by picking your battles.

First, Jeff and I had to come to terms with the existing code. As we settled in, we found we had to "move the silverware" in a few places to meet our tastes. In particular, we overhauled the look and behavior of the user interface. We left the reflection-based loading of classes in place, though.

Then there were the minor differences in working style that I noticed between me and Jeff. For the most part, these were so minor as to border on petty: "If I'd done it, I wouldn't have spaced the output that way" or "I usually write one class at a time rather than mock up all the classes and then fill in the details later."

A couple weeks ago, I mentioned the motivating aspect of working in a group. My point this week is that, whenever you work with someone else, they are not going to do things exactly the same way you do. That is to be expected. But, even when you realize this consciously, each specific difference you discover can still cause a short pause: "Oh.. that's not how I would have done that... but that doesn't mean it's wrong... Am I going to accept how this has been done and move on, or do I want to say something and change it?" Overall, I discovered that I could just let these things go. It was just a new experience for me to adjust to someone else's presence in my coding project space, both in terms of the old code and the new.

Jeff and I got everything finished up before the deadline. I think our code is pretty solid, though some of our line-spacing might be a little off between different commands.

Testing code that connects to a server continued to be a pain. I considered making some sort of mock client object, but there proved to be too many methods that would need to be overridden to make it worthwhile.

I also found that, when verifying someone else's code, it's easier to run through a manual test rather than plod through all their JUnits and make sure it covers all the necessary cases. While I think it's good to get some manual testing in there occassionally--especially since that lets you spot things like typos and weird formatting that a JUnit test is not going to catch--I think this is still not quite ideal. Manual testing can find bugs that the person's JUnits missed. But, if the code is correct, manual testing won't reveal that an appropriate JUnit test is missing altogether. This means that changes to the code could cause an undetected failure later. So I guess I should work on using the person's JUnit tests as a loose guide to manual testing in order to spot missing/poor tests as well as any existing defects.

I'm still a fan of Issue-based Project Management. I'm also glad we kept our habit of having the other person mark completed tasks as Verified. It's an extra step, but it's a good feeling knowing that everything has been double-checked by someone else.

This is the last project for my software engineering course, so it may mean a blog hiatus for a while. But I should be back occasionally with news on other new software projects!