I’ve been posting altogether too much non-technical stuff lately. That changes for the better today.

One of the first things I did when I joined AppAssure in the summer of ‘06 was stand up a bug tracking system. For reasons of expediency and cost, we went with Bugzilla, the open-source defect tracking system built initially to support the development of Mozilla. Bugzilla worked fine, but it had some pretty annoying limitations, not least of which was the generally klunky feel of the UI, and the trouble we had using it as a help desk system as well as a defect tracking system.

In Q4 of last year, I started looking around at alternatives, and I kept coming back to FogBugz from Fog Creek Software. I trust Fog Creek as a company due entirely to the seemingly endless supply of insightful blog postings from Fog Creek’s co-founder, Joel Spolsky, posting at Joel On Software. While I don’t agree with everything Joel says, I’ve nonetheless benefited immensely from his writings on project planning, hiring, and retention.

What finally sold me on FogBugz was the introduction of FogBugz 6.0 with its flagship feature, Evidence Based Scheduling. Joel wrote a long post on the idea behind EBS, which convinced me this was a feature I had to have. Another awesome development was the introduction of FogBugz On Demand, a hosted version of FogBugz run out of data centers and managed by Fog Creek personnel. That meant we could pay a monthly fee and let Fog Creek worry about the hosting details, which is definitely not our core competency.

Last January I migrated our Bugzilla bugs into FogBugz using a little Ruby script I wrote (about which more later), and since then we’ve been using FogBugz exclusively. Now that I’ve been running it for a while I feel qualified to write up a review of our experiences with the tool.

Cons

Migration from BugZilla is painful

When I was planning the switch to FogBugz, I relied in the FogBugz 6.0 docs, which state:

FogBugz ships with a script for importing bugs from Bugzilla into FogBugz. This file is called importBugzilla.asp on Windows systems and importBugzilla.php on Unix and Mac systems.

What they don’t say is that this is not available if you’re using FogBugz On Demand. Of course, Fog Creek will happily sell you its custom migration services for a few thousand dollars, but that’s it.

Not being one to curse the darkness until I’ve at least lit a candle, I wrote a quick Ruby script to parse the XML output from a Bugzilla search for all bugs, and migrate the information into FogBugz using the FogBugz REST API. This was far from a perfect solution, though, as the REST API wouldn’t let me preserve the opened/resolved/closed dates or the date/time stamps on the case notes. I posted about the script here, if you’re interested in the details.

EBS Is Not Ready For Prime Time

Based on my reading of the Fog Creek support forums and responses to my support inquiries, I’m pretty sure Fog Creek aren’t using EBS internally yet, at least not in the form implemented by FogBugz 6. Though this is purely speculation on my part, I base it on some simple observations:

• EBS doesn’t accommodate basic planning concepts like dependencies between tasks
• There’s no simple or elegant way to handle bugs with EBS
• There’s no way to tell EBS if a given developer is working on a project for some percentage of his work hours; only a global setting for all cases in FogBugz

I’m trying to get my team to use EBS for the next release of our product to see if it adds any value, and we’re doing well so far, but with the above deficiencies it’s hard to see how the shipping estimates EBS generates can be taken seriously for a project of even moderate complexity.

Sometimes KISS is the wrong answer

A striking difference between FogBugz and virtually every other defect tracking tool on the market is the dearth of data fields supported by FogBugz, and the relative simplicity of its features. This reflects a deliberate design decision made by Joel and others at Fog Creek, and it’s one I understand, but in a few cases, Keeping It Simple Stupid makes the system considerably less usable.

One example of this lack of functionality is the very limited options for managing cases. You can set the project to which the case pertains, the area within the project effected by the case, and the release in which the case is to be resolved. The possible values for the “area” field cannot be expanded except by an admin, which is a deliberate (and, imho, correct) attempt to limit an explosion of “area” values. However, this makes it very difficult to do any tracking or grouping of cases by other parameters.

An easy and sufficiently simple solution to this problem would be the implementation of tagging for cases. By allowing users to tag cases with opaque textual tags, then group cases by combinations of those tags, arbitrarily complex organization schemes could be implemented without an explosion of database fields. FogBugz desperately needs a feature like this. While one can always “star” a bug so it shows up in the user’s “Starred Cases” list, that’s an awfully limited bit of functionality, and one that isn’t shared between users.

Another glaring example of this problem is the Filters facility, in which users can create private or shared filters which can easily be invoked to view, say, “Cases assigned to me” or “Cases opened in the last week” or “Cases assigned to the 6.1 beta release”. That’s fine, but the filter criteria are extremely limited. What if I want to view cases in the 6.1 beta and the 6.1 gold release? Nope. How about cases in both the “Logging” area and the “Reporting” area? No. In fact, for every field upon which you can filter, the filters feature allows you to specify only one value. Most unfortunate.

Correspondence functionality could be better

Since we use FogBugz for our help desk system as well as for defect tracking, we take advantage of its ability to monitor mail on a mail account and suck that mail into a FogBugz case. When that happens, the email address from which the mail was sent is noted in the case as the “correspondent”, and you can send mail back to the correspondent with the “Reply” feature.

This is handy, but what would also be handy would be the ability for me or another developer to use email to attach stuff to cases, without showing up as the correspondent. FogBugz knows our email addresses, so it should be able to detect mail from an existing user and opt not to flag that user as a correspondent, and instead attach the message and/or file attachments to the case as though the corresponding FogBugz user had made the change in the UI.

Obviously there are potential security problems with this approach, but FogBugz could be configured to trust our mail server not to lie about the originating address, and mitigate the exposure.

Pros

SVN Integration

By adding a simple VB script to our SVN box, we’re able to log all SVN commits with FogBugz. Then, by adding a SVN property to our repository and using TortoisSVN to do our commits (which we do anyway), we can explicitly tie SVN commits to one or more FogBugz case IDs.

This might not sound like a big deal, but it is. Our automated builds use the SVN revision number of the source tree as the build number (in addition to major and minor version numbers). FogBugz lists the SVN revision numbers for commits associated with a case. That means I can fix a bug or implement a feature, its case gets assigned to QA, and QA can tell if the latest build has the fix or not by comparing the highest SVN revision number in FogBugz with the build number of the build he’s testing.

All that’s required to make this work is a shop with the discipline to note the case numbers associated with each commit. Fortunately, I’ve been relentless in training our dev team to do just that since the BugZilla days, and with this feature in FogBugz it’s now in the developer’s best interest to tie commits to cases, otherwise QA will come calling to figure out which build has a particular fix. Too easy!

Search engine

The search feature in FogBugz consists of a single text box you type in, like Google. If you just type text, it uses its full-text index of the bug database to search for matching text in a split second. You can also use search operators similar to Google’s to create a more structured query. Compared to the slow and painful search in BugZilla, this is a godsend. I can type the name of a customer, for example, and see all the support cases opened by that customer. Or I can type a funny error message I remember seeing a while back, and get all the helpdesk and bug cases pertaining to that message.

I will never go back to shitty BugZilla searching.

Sometimes KISS is the right answer

Though I have complained about what I see as misguided application of the KISS principle in FogBugz, there are instances in which Joel and company made the right call. The striking lack of fields in a case, and the ease with which a minimal case can be opened, are points in FogBugz’s favor. There are elements within my company who would run wild with a system with a highly-configurable schema like ClearQuest. Having previously worked with a CQ installation with a multitude of tabs and special-purpose fields and onerous minimum entry requirements, I very much appreciate the FogBugz “less is more” philosophy.

Wiki

The Wiki functionality in FogBugz 6 is very handy. It gives us a central place to put docs like specs and guidelines that doesn’t require a VPN to access. Our QA guy puts his test reports there so we can all see which builds have completed which level of testing, and I maintain a wiki page with various “how to do things in fogbugz” guides to remind devs and support staff how our process works.

EBS Is Cool

As half-baked as EBS is, its current implementation is not without value. By following Joel’s advice an imposing a max estimated task duration of 2 days, my developers and I are forced to thoroughly think through our implementation before spec’ing it out, and the tracking of time against a case adds some accountability to the development process without getting in to clock-punching minutiae. I hope the next release of FogBugz extends EBS to the point where it’s actually usable on a moderately-sized project.

Easy case linking

In FogBugz, you can link cases by referencing one case in the case notes of another. Let’s say I’m adding a case note to case 1, and I type “This is the defect corresponding to helpdesk case 2”. FogBugz will parse out “case 2”, make it a hyperlink to case number 2, and both cases will have a link to the other case under “See Also”. This comes in very handy when we have one case opened by a customer to track a support issue, and another case to track the software defect which was discovered as a result of the support case. This way dev and QA can go through our normal fix process, and support can hold the support ticket open during the fix process and easily link over to the defect case to determine its status.

Conclusion

All bug tracking systems have some limitations, even the ones hand-rolled by project teams with acute NIH (Not Invented Here) syndrome. FogBugz is no exception. However, I feel comfortable paraphrasing Churchill here: FogBugz is the worst defect tracking system there is, except for all the others. There’s plenty not to like, and there’s plenty to disagree with, but in the end if I had to choose a single system I’d choose FogBugz (and, in fact, I did).

UPDATE: Read about my experience with Fog Creek’s astoundingly proactive support.