Bug Reporting¶
Steven Frank offered advice for reporting bugs. Just as Brent Simmons’, or Apple’s text on the issue did. While these texts are all correct, I found - and keep finding - that they represent a terribly biased point of view: a developer’s.
While a user may understand that such a lovingly written report may be extremely helpful to the developer, the user may also have a bunch of good reasons to not write a report in such detail. I will highlight a number of those reasons below and also try to see how different bug handling styles go with them.
Reason 1: Your software is full of bugs
All developers know their software has bugs or weaknesses. And most reasonable ones (i.e. most people outside PR drone dominated corporations) will admit that. These bugs cover the whole range from kernel panics for system level stuff, to application crashes, to data destruction, to details like odd behaviour during drag and drop situations, to preference loss, to poorly done localisations.
Perfect software will not exist once a certain level of complexity is passed and people seem to have accepted that. If one reported all bugs seen in everyday software use and did so in full detail, one wouldn’t be able to do anything else. For the rest of one’s life. Hence only the most annoying problems will be reported. Each report costs the submitter time. Thus he or she cannot give all the details the developer may want to have.
Reason 2: It is your software
I probably started using your software because I didn’t want to write my own or get involved in some open source project. Instead I wanted a black box that I need to neither understand nor care about. It’s supposed to be magic and just do the right thing. You provide that magic and get the credit for it. Likewise, the problems are yours as well.
Reason 3: The effect of a bug report is unclear
With bug reports it is completely unclear how much effect they will have. In my experience the range goes from detailed bug reports which were not acted on or even acknoweledged after years to quickie bug reports for which I received an updated beta version within hours. Unless you have confidence in the developer who receives the bug report, it seems silly to put a lot of effort into an initial submission.
Reason 4: It is not easy to make bug reports
I will submit a bug report because I ran into a problem. If actually submitting the report is more work than tolerating the bug is, the reporting won’t happen. Bug reporting systems come in a wide range of qualities. Most of them are horrible and totally inconvenient and manage to put up so much red tape that the bad mood your bug put me in has become much worse by the time I managed to report it.
These are a few obvious reasons. And it’s entirely within the power of developers to reduce their impact. Unfortunately very few developers use that opportunity.
Solution 1: Ship less buggy software
Haha, obvious one! While I want to naïvely believe that most developers are not BOFH types who intentionally code bugs into their software, this belief is obviously wrong. Effectively many software registration and all DRM schemes are just huge and intentional bugs which have the ‘collateral damage’ of keeping paying users from using the software (this covers the whole range, just speaking to anybody in charge of a room of machines running Adobe products usually provides examples but even on the small end I have seen self-destructing NewsFire licenses which required an internet connection to be revived).
In addition to those intentionally implemented bugs it also seems like people will consider shipping software with accidental bugs they know of to meet schedules and so on. I guess that this is pretty much standard practice and in part crucial to getting the software out of the door.
More bug fixing, better testing and more realistic schedules will certainly improve the software in that respect. In all cases it might be advisable to simply document the known problems in a way that is obvious to the user. That will save the user the agony of running into the problem. Many buggy areas of software are less problematic if you know how to avoid them. People who want to meet their deadline with a non-trivial Word document won’t change their printer settings at the last minute; people needing reliable FTP connections won’t use the Mac OS X Finder for that, and so on.
Devlopers will have a list of known bugs and problems. Making that list available has two advantages: It will save people the effort of submitting a bug report. And it will show that the developer is admitting there is a problem which can be seen as a commitment to solving it in a reasonable time frame. Both things can leave a good impression with users.
Solution 2: Decommercialise
Go open source. If you want to be able to tell your users to go, read, and fix the source code to remove the problem themselves, that’s totally the way to go.
You probably didn’t want to go that route but rather use software as a way to pay for mortgages and college funds, so now all those bug reports are yours to enjoy as well… If you want to pretend you’re an actual software developer who likes helping people, you’ll have to do a bit better than open source and let your users to be just users with no huge time commitment to improving your software.
Solution 3: Be transparent
Here you can earn the trust of your users by sharing the information you have. Make it easy for users to find known issues in a human readable form (i.e. Bugzilla &c do not count). Make sure the people who report issues will know their report has been received and make sure they can follow up to that in case further details emerge.
Also let bug reporters know about unclarities in their reports or, in case you do not intend to change the reported behaviour, just tell them. It’s always better to know what not to expect than to be disappointed release after release (Just imagine Apple had publicly stated that they intended the Finder to remain a mess when releasing Mac OS X.1, nobody would have been surprised since and there might even be a good replacement by now).
Solution 4: Make reporting bugs easy
This cannot be reiterated frequently enough: Most bug reporting processes are a royal pain. Instead they should be a pleasure. Frequently seen failures are:
- Systems requiring a login
- Systems not accepting submissions by e-mail (i.e. being web-only and thus requiring me to have a working internet connection when typing the report and making it difficult to attach files)
- Systems not sending you a copy of the report you filed together with the case number (in case there are case numbers) for future reference and easy searching
- Systems not giving you a case number immediately (meaning you can’t send along further attachments right away)
- Web components of the system running on slow servers
- Web forms with ‘required’ fields that prevent submissions from succeeding (I doubt that filling out ‘EXPECTED: no crash … ACTUAL: application crashed’ helps anybody).
- Systems requiring you to refer to attached screenshots by name rather than letting you drag them in the position where they are needed while writing the report.
Many of these problems come from the fact that bug reporting software is apparently a genre which isn’t designed for usablity or convenience. And many developers - people who may actually appreciate receiving nice reports - don’t seem to see this as a problem. If you think people should include some information about their system in bug reports, give them a one-click menu item to create an e-mail with that information. If you need crash reports or screenshots, tell people about it and prepare as much of the information as you can without user intervention.
Expecting users to manually collect all the relevant information which a machine can easily generate for them automatically while also expecting them to send convenient and well formed reports seems wrong to me. Completely wrong. Cognitive dissonance.
Of course many of these issues merit further and more detailed discussion: Which information can be sent? how should the users’ right for privacy be handled (stuff like X.5 hang logs or system profiles seem to contain a load of information which leaves you with a bad feeling; your bug reporter has to make clear who can read the cases mentioned there along with the log messages or other data users upload)? Who will write the ‘Sparkle for bug reports’ framework? And who’ll lobby Apple into making capturing films of the screen (something I consider a huge time saver for bug reports because you can demonstrate the problem) easy - complete with a reflection of mouse movements, clicks and keypresses?
Who’s reading it
Another qeustion is who reads the bug reports. The larger the software company, the lesser their competence at getting the right people to see those reports. Be it because they don’t read them at all or because the reports are first inspected in some way by support staff who don’t necessarily know what they are doing. Occasionally one gets the impression that the people reading a report have no clue. Such formalised systems also have the disadvantage that you may actually have a better understanding of the problem and of troubleshooting techniques than the person processing it. Which is exactly the frustrating telephone hotline situation you alienate people with. This may be non-trivial for a greedy company, but getting qualified staff in those positions will help.
I’ll finish with a few examples of how bugs are handled by Mac developers. Perhaps people can find a good idea or two in there and steal it.
Apple
The corporate monolith in the Mac universe. There are many ways of getting bugs towards them. The operating system itself offers to send crash, hang and kernel panic logs straight to Apple. As that is a no-effort proposision, it is probably a good idea to do that (at least for crashes, those hang log files seems to contain a load of information that goes far beyond the application which is hanging; it’s also unclear to which extent sending in crash logs for non-Apple applications will make any difference - quite certainly the developer who should see it will never even learn about it). The general guess seems to be that those logs - just as the well hidden but in some sense public bug reporter on Apple’s web site are mainly a source of statistical entertainment for the company. Putting any effort into such reports is quite likely a waste of time.
Apple have other bug reporting systems as well - for example with their developer site - and at least the serious reports like kernel panics seem to be looked at there. Less ‘serious’ issues are a mixed bag. Some are ignored for years, a few are fixed, others are classified ‘behaves as specified’ while most end up being duplicates. With those duplicates obviously marking another unattractiveness of closed bug reporting systems as they mean that more than one person wasted their time pointing out the problem.
The Mac universe seems to think that filing duplicate reports of the same bug somehow improves the situation. I consider that to be wrong. Even if filing the same thing many times gave better results, it’d ultimately be a waste of time, both for the users and for the people processing the reports. It’d also hint that Apple’s bug processing is very poorly managed and that any e-mail from them calling you a ‘valued’ customer is a farce.
Karelia
Karelia put quite a bit of effort into their bug reporting system by now. Sandvox comes with its own feedback reporter window which lets you do the things you typically need to do:
- submit anonymous or named feedback,
- receive a copy of your feedback by e-mail,
- not give you dozens of red-tape fields,
- automatically include the version/plugin information of Sandvox in the submission,
- provide a reminder of the preferred format for feedback,
- read a promise that your report is kept private,
- offer to send along console messages, screenshots and settings and
- follow up on previously submitted problems.
To top things off it also doesn’t fail and die when you don’t have a network connection at the moment you use it, but it saves the report and sends it the next time that’s possible. In addition to that Sandvox automatically discovers crash logs when the application launches and offers to send them in.
More trusting users can turn on automatic submissions of such reports - as well as of exception reports which are only sent once in case the same exception appears several times - which, particularly when using betas, can relax the situation significantly as you know the application will file whichever information is needed without any effort of yours. And if you want to track things or submit additional information you’ll find the relevant case number in your console log.
This system still has a few rough edges but it’s certainly a very good step towards making filing problem reports as painless as possible. In addition to this, you can still submit reports to Karelia via e-mail as they seem to be using FogBugz which appears to provide a reasonable e-mail interface (we better don’t start talking about their table infested web pages which quickly reassure you of their Microsoft DNA).
Panic
The beauty of Panic’s problem reporting system is that it’s completely informal and works via e-mail. That’s step 1 and it is very convenient. The second - and I believe equally crucial - step is that their bug reports are processed by intelligent human beings like Les who do a great job at pulling whatever formal strings need to be pulled in the background and keeping the reporter informed. All this is done completely informally without any numbers or other red tape attached which gives it a very good ‘feel’ even though it may not completely satisfy your inner control freak.
Personally I rather like that style of bug processing because I really don’t want to track the problem I submitted. I observe the problem and if it annoys me enough, I’ll submit it. Then my job is done and the recipients have to figure out what I mean, whether they want to fix it and when they are going to do that. Only afterwards I need to learn about the outcome of this as I can’t influence the rest anyway. And if one can trust the developers to be reasonably good at not losing or ignoring cases just because they didn’t give you a tracking number for them, that’s probably even better than giving you a tracking number.
For beta testing Panic also tried out more formal bug tracking system (Redmine). It is web-based, open source and thus - unsurprisingly - inconvenient and sucky.
Lemkesoft
Many people sneer at GraphicConverter because it never quite made it to OS X UI-wise. That’s a justified critique, but at the end of the day GraphicConverter gets the job done and - unlike many younger and hipper applications - still supports technologies like Mac OS 8 and 68K processors. So I am tempted to think that this is more about having a different focus than about a lack of effort.
In fact, Lemkesoft’s bug report and support system is all about making an effort. For many small problems which I reported over the years - using the e-mail link provided in the application -, I frequently received replies, requests for further details or even beta versions for testing the fix within short time spans; Sometimes within hours. All without case numbers or formalities.
I suppose this way of working doesn’t really scale to larger companies, but it’s still the best experience you can get.
