Group Programming and Issue-based Project Management

I spent the last couple weeks doing some group-based programming on a small WattDepot project. The project was to build a command-line interface that lets a user poll the energy data collected from one of the dorms on the University of Hawai'i--Manoa campus. Different commands allow the user to see the current power consumption of a dorm tower or lounge, to rank the different dorm towers by energy consumed over a given time period, to see the energy use of a tower or lounge for a given day, or to see the total energy consumed by a tower or lounge since a given date.

I haven't done any group programming in about 10 years, but I found the return to it quite enjoyable. I find it motivating to know that someone else depends upon and is eagerly awaiting my changes. I found myself prioritizing this project over my other projects to make sure I wasn't holding someone else up.

Our group consisted of only two programmers. We used Google Project Hosting's issue-tracking system to manage the project. We met occasionally to map out the course of the project, and we would create new issues for tasks that needed to be done. Then one of use would accept responsibility for the task and work on it. Once it was done, the other person would look over it and mark the issue as verified. Since the different issue pages allow for threaded discussion, I found this is a good way to track the progress of the project. It was also very easy to link from source code commit logs to the relevant issues and also back from an issue to those commits that solved it.

We happened to worked in shifts, and so we never had to deal with a code merge. I'm a little disappointed I haven't had a chance to experience this yet.

We also used continuous integration. That means we had a Jenkins server setup that independently verified that each commit to the Google repository still compiled successfully and passed all tests and QA checks. This was a comfort. After a commit, I'd always wait a few minutes to make sure I didn't receive an email from Jenkins. When I didn't, I felt more confident signing off for the night.

Our project was plagued with external problems, though. I had to make a emergency 5-day trip back to the mainland for a funeral during the middle of the project. My partner, Jeff, broke his leg around the same time. Besides being on painkillers after that, he wasn't able to drive to campus. So we only had one initial face-to-face meeting, but we compensated with a couple phone meetings and nearly-daily overview emails in addition to the issue-tracking threads. Near the end of the project, the WattDepot server went down, which halted production and then broke all of our test cases. This meant that our build appeared broken for about 3 days while we waited for the new server to generate enough data for us to work with.

All this planning and coordination certainly did come with an overhead. I'd say this project took 2 to 3 times longer than if we'd just done the project independently. But, like I said, I found the group atmosphere quite motivating, and it probably helped to have a couple pairs of eyes look over all the code.

WattDepot Katas

So I spent part of this week and all of this evening getting up to speed with WattDepot by completing a few practice katas. Each kata took me about an hour, though for different reasons.

The first couple katas were simple requests from a WattDepot server.
Given the sample code from the WattDepot client tutorial, this required only a big copy-and-paste and the modification of a couple lines.

These still took a couple hours, though, because I did the first couple katas in class on a day that we tried pair-programming. I think this might be a good practice once both programmers are comfortable with the API and understand the problem at hand. As it was, we spent most of the time setting up our Eclipse projects and then reading the assignment and the WattDepot documentation over each other's shoulders. I also switched groups halfway through, and we were also required to swap laptops periodically. This means the second pair I was in ended up starting over again at the beginning of one of the katas on a different machine.

Lulled into a false sense of security that these katas would all be as easy as the first couple, I returned to the project late this afternoon. I first took some time to install the FindBugs plugin in Eclipse. (I've been gradually installing plugins for tools that I have so far been using Ant to run. It's generally tedious and slow to run Ant compared to a plugin. But it's been a slow transition since I like to spend an hour or so reading the documentation and playing around with each plugin before I install the next one.)

After cleaning up what code I threw together in the last few minutes of class to see it work, I finished up kata #2. That kata and the ones after it all require sorting a list of key-value pairs by value. This stumped me a for a little while because this should be very easy... but it's not. I surfed a few 1/2 to 1-page posts of code that sort a Map by value. There were a few nice one-liners, but they required Google Collections and I didn't want to mess with an extra library.

In the end, this is my solution to sorting a key-value list by value. It's conceptually a "two-liner", but all the generics and the anonymous class make this look horrible:

  Map<String, Long> map = ...  //the map to be sorted by value

  //dump map into a list of key-value pairs...
  List<Map.Entry<String, Long>> entryList = 
      new ArrayList<Map.Entry<String, Long>>(map.entrySet());
      
  //...and sort it with a comparator that sorts based on values
  Collections.sort(entryList, new Comparator<Map.Entry<String, Long>>() {
      public int compare(Map.Entry<String, Long> left, 
                         Map.Entry<String, Long> right) {
        return left.getValue().compareTo(right.getValue());
      }        
  });

  //now print the list in value order
  for (Map.Entry<String, Long> entry : entryList) {
    System.out.print(entry.getKey() + ": " + entry.getValue());
  }

This solution also assumes you don't need map-like access to the sorted version, but just need to run through it for a simple task like printing.

The other time-consumer was learning how to use java.util.GregorianCalendar and
javax.xml.datatype.XMLGregorianCalendar. (There's also org.wattdepot.util.tstamp.Tstamp,
though I didn't find this to be as useful as I'd initially hoped.) This was another example
of something that should be easy but isn't. For example: "Give me the date of the most recent
Monday". It seems we have to start with the current time, for which I chose to use a GregorianCalendar object. Then, after a few lines spent modifying that time to the desired relative Monday date,
I had to translate it to an XMLGregorianCalendar. This was fairly long and messy with the various required
factory objects and exception handlers. I still like Java a lot, but I must admit that detractors
have a point about how much grinding tedious boilerplate you have to wade through sometimes.

I also need to take a minute at some point to figure out how to control Eclipse's insertion of try/catches. The quick default just wraps the one line in a try block, which requires so much reformatting that the feature is hardly worth using in most cases.

In the end, I got all the katas done. This was the first time I've written an application that deals with a web connection behind the scenes. It is pretty slow and variable (read: flaky) at times due to network access delays and potential connection time-outs. For my last kata--which took a couple minutes to gather all the data it needed--I added a little "status indicator" by printing a series of dots.

I spot-tested my katas as I went. However, if they were production code, I think I'd like to test them a little bit more to make sure I'm really pulling the right data and that all my calendar manipulations worked correctly. But since they are mainly katas intended to practice solving the problem, I think they went quite well.

Energy Consumption in Hawaii

I'll soon be working with WattDepot, so this week I've been learning a bit about energy consumption in Hawaii as background.

First, I had to review bit of high school physics regarding the difference between energy and power. Power is rate at which work is done or energy is converted over time. Power is measured in watts. One watt is equal to one joule/second. So a 60W lightbulb burns 60 joules/second.

Energy is... well, energy--the stuff that makes things move, go, change, or happen. It is force applied over a distance. When related to household electricity, it is usually measured in kilowatt-hours, or kWh. One kWh is equal to 3.6 megajoules.

I also learned that Hawaii is in a pretty unique situation when it some to exploring alternate/green energy sources. Our energy on Oahu costs about 30 cents per kWh, which is 3x what it costs on the mainland. (And on the neighbor islands it costs 6x mainland prices!) This is because we rely predominantly (70+%) on shipped oil, whereas the mainland can also rely on nearby coal and natural gas sources. We have a number of small, discrete electrical grids here, rather than the massive interconnected grid that spans much of the mainland. This means we can't buy, sell, or shift extra energy around to neighboring states (or even neighboring islands) as demand shifts. This is fairly inefficient.

But, on the bright side, this expensive energy means that alternative energy sources may be more attractive here in Hawaii than they would be on the mainland. The Hawaii Clean Energy Initiative has also mandated that 70% of Hawaii's energy come from clean sources by 2030. To our benefit, we have nearly every kind of potential renewable energy source here: wind, solar, waves and tidal motion, hydrothermal, and geothermal sources.

Data about current energy use and how this usage will be affected by alternative sources will be essential. This information can also provide feedback to end-users to help them reduce their consumption, which is also an important goal. There is currently some interesting research going on at the University of Hawaii regarding smart sensors and tracking current use... which brings us back to WattDepot. It is currently being used to track energy use in some of the UH dorms.

I'll be learning more about the specifics of WattDepot this week. Check back again soon for more info!

Collaborative Review

The students in my software engineering class are reviewing for the midterm.
To contribute to the group study session, here are 5 review questions (and answers)
of my own:

1) Name three features often provided by an IDE.

The most common features include:

• Language-aware editing (syntax-highlighting)
  
• Integrated compilation (and execution)
  
• Integrated debugger
  
• Project definition facilities
  

Extra features may also include: wizards (class creation, etc), content assistance, refactoring, diagramming, project file manager, plugins (for tools such as configuration management, QA testing, JUnit, etc).

2) Give an example of an annotation tag commonly used in Java.

The Java compiler recognizes: @Override, @Deprecated, and @SuppressWarnings.

JUnit uses: @Test, @Before, @After, @BeforeClass, @AfterClass, and @Ignore.

Annotations can also take parameters, as in: @Test(expected=UnsupportedOperationException.class).

3) Which two methods in class Object should you override if you suspect that instances of your class might some day be need to be stored in a Collection?

equals and hashCode.

4) How do you use @Override in Java? What does it do?

@Override is an annotation that you should add at the beginning of a method definition if that method is intended to override a method in a superclass or interface. Example:

  @Override
  public String toSting() { ...

Thanks to the @Override tag, I would now get a compile-time warning that toSting() does not actually override a method. (This is because I misspelled String.)

Using @Override when overriding interface methods is only supported with Java 1.6+.

5) Ant is a build system. Name another one.

Maven and Make are also build systems.

Project Hosting: Hosted for the Very First Time

This week added another tool to my programming toolbelt: project hosting and software configuration management.

When working on my dissertation project, I used Subversion (SVN) locally to keep track of different versions of the project's source code. This meant I didn't have to worry about making a big implementation mistake because I could always rollback to a previous verion. Since I was the only programmer, I only committed weekly. The commit notes made for a good log of each week's work.

But this week, I setup a test project at Google Project Hosting for my BusyWeek robocode robot.

Setup went pretty smoothly. Google provides a pretty standard and simple interface to all their services, so nothing was too surprising there.

The built-in wiki and bug-tracking tools are pretty nice. (I haven't had a chance to explore the bug-tracking yet, though.) The wiki markup can be annoying--but isn't that true of every wiki system? Personally, I found the requirement to indent every * list bullet by two spaces rather annoying. And I couldn't seem get my numbered list to continue if I included a code block within one of the points. While all the various wiki markups are supposedly easier, I find I prefer HTML in these cases because then I have clear and explicit control. But this was all pretty minor stuff. The Preview button was a lot of help here, so that I didn't fill the revision history with minor edits.

It's also a little strange that editing a wiki changes the revision number on your software. However, I suppose the contents of the wiki and the contents of the code base are highly related, so this approach probably provides a clearer project history.

I did get side-tracked in selecting a license, though. I still haven't found a simple license that I really like, especially among the OSI licenses. Since my Ant build files and one of my test files is based on a DaCruzer robot project, I figured I had better use the same license. (This is fine, since I like the Apache 2.0 License.) However, neither my project or DaCruzer actually includes any of that license information in the source code itself. I should probably look into that at some point.

Anyway, it's exciting to actually have a project hosted somewhere now! Fun stuff.

BusyWeek: Programming Under Pressure

It's been a busy three weeks! I completed two programming projects in that time (as well as a lot of other work). The contrast between how those two projects went provided me some interesting insights on testing and general best-practices.

Three weekends ago, I completed a hefty implementation project for an algorithms course I'm taking. I had to implement and gather runtime data on 4 abstract data types: a sorted doubly-linked list, a skip list, a binary search tree, and a red-black tree. All of the implementations adhered to the same interface. I then had to write a text-based user interface to allow a user to run performance tests as well as call methods across all 4 implementations. All told, the project took me about 30 hours over 3 or 4 weeks--though half of that time was put in over the last two days. But the project was a success. (I had one known bug at the midnight due date, but I was able to fix it and resubmit an hour later.)

I think part of that success was in how I spent my time. I started early and spent that time designing the common interface, implementing the easiest data type (linked list), and writing a single battery of JUnit tests that I could then run across all 4 implementations. When crunch-time came, I was confident that my tests were robust. I could just focus on the implementation. When all my tests passed, I felt I was on pretty solid ground. In short, test-driven development eased that last-minute stress.

My other recent project was to design and test a robocode robot.

My original design for this was named JiggersTheFuzz. (The name is an obscure reference from an old Sierra game named Police Quest.) I planned to use linear targeting, which means anticipating a target bot's location by leading their current position by an amount relative to their velocity. I planned to scan enemies for energy drops that mean they'd just fired and then try to dodge their bullets (the "jigging" of the name). If robocode bots want to "mark" another bot, they often use the getName() method. So I wanted to dig into Robocode and the runtime stack a bit to see whether I could override my own getName() to return a different String each time but in such as way that it didn't mess up the robocode simulation itself. This would be like a radar jammer (the "Fuzz" of the name): other bots would still see me, but they wouldn't be able to identify me between turns.

However, it was not to be.

Two weekends ago, I had a conference in California. While I managed to get caught up on everything else before I left, this robocode project followed me onto the plane. I tried working on it in my hotel at 9:30pm after two sleep-deprived nights and a long conference day. I tried again at 5:30am. It just wasn't happening.

I had a plan sketched out, but the details still involved too many unknowns. In short, I hadn't started early enough to be on solid ground come crunch-time. I started paring away features as the deadline neared. The first to go was the radar "fuzzing" (so I never got to look into that one). The second to go was scanning other bots to track their energy level. (This is actually very hard to do in robocode when extending Robot rather than AdvancedRobot, since a Robot can only do one thing at a time: turn its radar, move, or fire, but not at the same time.) I was down to just moving regularly in the hopes of dodging bullets and generally being unpredictable... but I wasn't handling wall collisions at all. There was little left to pare away and still have a functional bot.

At this point, I called it: This project is going to be late.

So I renamed it BusyWeek. I kept the linear targeting and the naive bullet-dodging, and, once I got back to my normal life, I sat down to do it right. Last weekend, I finally finished my bot.

Movement: BusyWeek assumes 1-on-1 combat. Once I scan an enemy, I turn to be perpendicular to him. Then I move 20px at a time, which is a touch more than the width of my bot. Thus, if he happens to shoot just before I move, I'll be out of the way by time the bullet arrives. Of course, this assume he's at least 140px away and firing small fast shoots. (He can be only 77px away if firing big slow shots.)

To ensure I'm this optimal distance away, my bot tries to stay between 140px and 180px away. (Farther than that, and it's too hard to make my own shots reliably.) If I'm currently outside of this optimal range, I'll turn to 45 degrees (instead of 90) from my enemy's heading and move that way. I handle walls by moving to the opposite side of the enemy's heading to avoid wall collisions. If I've been backed into a corner and can't avoid a wall collision in either direction, then I try to ram the opponent instead.

Targeting: As mentioned, I used a form of linear targeting to lead the enemy based on his current heading, velocity, and distance from me. I decided to include a "fudge factor" of treating his velocity as one less than it actually is. This was because it seems that stops and turns are more common behaviors than continuing along in a straight path.

Firing: BusyWeek shoots bullets with a power proportional to its own current power level. This way I don't shoot to the point of becoming disabled. Since smaller bullets move faster, it also means I hopefully become more accurate as things get more desparate. Also, I don't even aim at an enemy if my gun is still too hot to fire. This saves me some time for scanning and moving instead.

This design worked fairly well against the standard sample bots. Here are my win rates over 100 non-deterministic rounds each:

vs sample.SittingDuck: 100%
vs sample.Tracker: 79%
vs sample.Corners: 85%
vs sample.Fire: 99%
vs sample.Crazy: 100%
vs sample.SpinBot: 67%
vs sample.RamFire: 60%
vs sample.Walls: 76%

Testing this bot proved to be a trial, though. I used a number of computational methods--such as finding a point on the board based on a starting point, a heading, and a distance. Testing these using JUnit tests went fine. I only wish I'd written the tests first. I actually wrote most of these methods for some of my kata bots weeks ago, and so I was already confident they worked. This takes all the fun out of test-writing.

Testing the behavior of my bot during the simulation proved to be trickier. For example, early in my design I decided that it'd be easy to test that my proportional-bullet-power feature was working correctly. (Testing this for bugs was a bit like only looking for lost keys under a streetlight--not because I expect to find them there but simply because searching in the light is much easier!) I simply paired BusyWeek with a bot that always fires shots of power 3. Then I used the RobotTestBed to snapshot all the bullets in the air each turn. I figured if some of the bullets decreased in power as the battle progressed, then it meant my algorithm was working. However, while the test passed, when I set my bot to only fire shots of power 3 too, I couldn't make my test fail! After some digging, it turns out that a bot--even a bot like SittingDuck that never fires--effectively fires a single shot of power 1 when it explodes. So I figured I'd just exclude all bullets of power 1 from my test. But then I realized that sometimes the other bot would fire lower-powered shots near the end of the match when they didn't have enough energy left to make a full-powered shot. Sorting through all these details took over an hour--effectively debugging the flickering of the streetlight rather than looking for my keys. Finally, I gave up: this was all to test a single line of code that I could just eyeball!

So, in conclusion, I learned the following:

• Start early and get the design and necessary research out of the way. This will let you make a more realistic estimate of how much time the rest will take you.
  
• Write tests before you write code. (Or, at very least, write them together.) Writing tests after the fact is drudgery.
  
• Write code so as much as possible can be unit-tested. Trying to test emergent or complex behavior through a simulation or an user interface is much harder.
  
• Make sure your test can fail before you rely on it when it passes.
  
• When you're at a conference, you're not going to have either the time or the energy to work on anything else.
  

Admittedly, these lesson are all very standard software engineering tips. And I knew most of them, at least consciously. But it's nice to learn viscerally that they really are valid!

Ant Katas

So I'm continuing to pick up programming skills. This week, I played around with Ant.

Ant is a build system for Java projects, like make is for C projects. Like all build systems, its primary function is to let you write scripts that intelligently compile the source code files in your project. But it lets you do a lot more than that. Directories referenced in a build script can be relative to the project's directory, meaning its easy to transport the whole project to a different system without having it break. Ant handles any differences across OS environments, such as whether the local system uses \ or / as the path separator. You can write scripts to automate other tasks, such as running test cases, executing other tools like as javadoc, or cleaning up stray files. The scripts can also test conditions in the local environment and adapt their behavior accordingly. When combined with Ivy, Ant can also do dependency management for you, downloading any 3rd party libraries required by your project. In short, Ant lets you automate a lot of routine programming tasks for a specific project. And it does this in a transportable way, across OS environments, so that anyone else who downloads your project can perform those task out-of-the-box too.

To get some practice writing Ant scripts (which are specified in XML), I went through a few katas for a simple Hello, World program. The Ant tasks involved compiling, cleaning, generating javadocs, and packaging the system for distribution.

What I learned from this is that the Ant Manual takes a bit of getting used to. It provides a decent tutorial, but it can be hard to use as a reference. For example, the manual includes a extensive list of all the possible task tags that Ant supports, including documentation of all the attributes they support. But for a while, the only place I could find a link to this important list was an in-text link buried in text of the "Using Ant -> Tasks" manual section. (In writing this post, I found there is a more direct link under Ant Tasks in the Table of Contents, but this is easy to overlook because its location between "Running Apache Ant" and "Concepts and Types" makes it look like a tutorial chapter title.)

I had a bit of trouble using Eclipse write these scripts. Eclipse has a special Ant editor which provides some helpful syntax highlighting and content assistance. However, I couldn't find a way to create an Ant file directly. Instead, I had to create a normal XML file, then right-click on it and open it in the Ant Editor (which required some browsing to find among the various editor options). Once opened there, it was possible to just hit the run button to run the Ant script, which was handy. And I only had to do this once for each file; after that, Eclipse remembered that it was an Ant file.

So I've covered the basics of Ant at this point. I'd like to mess around with filesets and paths a bit more. When I get a minute, I'd also like to figure out how to exclude Eclipse contents (such as the /bin directory and .project file) from distribution zips I build with Ant. Then, since I'm still learning Eclipse too, I want to see how Eclipse handles importing projects that don't have any Eclipse metadata included.