Quarter Life Crisis

The world according to Sven-S. Porst

« Symmetries ScriptingMainBlitzschutz »

Bug Reporting

2691 words on ,

Steven Frank offered advice for reporting bugs. Just as Brent Simmons’, or Apple’s text on the issue did. While these texts are all correct, I found - and keep finding - that they represent a terribly biased point of view: a developer’s.

While a user may understand that such a lovingly written report may be extremely helpful to the developer, the user may also have a bunch of good reasons to not write a report in such detail. I will highlight a number of those reasons below and also try to see how different bug handling styles go with them.

Reason 1: Your software is full of bugs

All developers know their software has bugs or weaknesses. And most reasonable ones (i.e. most people outside PR drone dominated corporations) will admit that. These bugs cover the whole range from kernel panics for system level stuff, to application crashes, to data destruction, to details like odd behaviour during drag and drop situations, to preference loss, to poorly done localisations.

Perfect software will not exist once a certain level of complexity is passed and people seem to have accepted that. If one reported all bugs seen in everyday software use and did so in full detail, one wouldn’t be able to do anything else. For the rest of one’s life. Hence only the most annoying problems will be reported. Each report costs the submitter time. Thus he or she cannot give all the details the developer may want to have.

Reason 2: It is your software

I probably started using your software because I didn’t want to write my own or get involved in some open source project. Instead I wanted a black box that I need to neither understand nor care about. It’s supposed to be magic and just do the right thing. You provide that magic and get the credit for it. Likewise, the problems are yours as well.

Reason 3: The effect of a bug report is unclear

With bug reports it is completely unclear how much effect they will have. In my experience the range goes from detailed bug reports which were not acted on or even acknoweledged after years to quickie bug reports for which I received an updated beta version within hours. Unless you have confidence in the developer who receives the bug report, it seems silly to put a lot of effort into an initial submission.

Reason 4: It is not easy to make bug reports

I will submit a bug report because I ran into a problem. If actually submitting the report is more work than tolerating the bug is, the reporting won’t happen. Bug reporting systems come in a wide range of qualities. Most of them are horrible and totally inconvenient and manage to put up so much red tape that the bad mood your bug put me in has become much worse by the time I managed to report it.

These are a few obvious reasons. And it’s entirely within the power of developers to reduce their impact. Unfortunately very few developers use that opportunity.

Solution 1: Ship less buggy software

Haha, obvious one! While I want to naïvely believe that most developers are not BOFH types who intentionally code bugs into their software, this belief is obviously wrong. Effectively many software registration and all DRM schemes are just huge and intentional bugs which have the ‘collateral damage’ of keeping paying users from using the software (this covers the whole range, just speaking to anybody in charge of a room of machines running Adobe products usually provides examples but even on the small end I have seen self-destructing NewsFire licenses which required an internet connection to be revived).

In addition to those intentionally implemented bugs it also seems like people will consider shipping software with accidental bugs they know of to meet schedules and so on. I guess that this is pretty much standard practice and in part crucial to getting the software out of the door.

More bug fixing, better testing and more realistic schedules will certainly improve the software in that respect. In all cases it might be advisable to simply document the known problems in a way that is obvious to the user. That will save the user the agony of running into the problem. Many buggy areas of software are less problematic if you know how to avoid them. People who want to meet their deadline with a non-trivial Word document won’t change their printer settings at the last minute; people needing reliable FTP connections won’t use the Mac OS X Finder for that, and so on.

Devlopers will have a list of known bugs and problems. Making that list available has two advantages: It will save people the effort of submitting a bug report. And it will show that the developer is admitting there is a problem which can be seen as a commitment to solving it in a reasonable time frame. Both things can leave a good impression with users.

Solution 2: Decommercialise

Go open source. If you want to be able to tell your users to go, read, and fix the source code to remove the problem themselves, that’s totally the way to go.

You probably didn’t want to go that route but rather use software as a way to pay for mortgages and college funds, so now all those bug reports are yours to enjoy as well… If you want to pretend you’re an actual software developer who likes helping people, you’ll have to do a bit better than open source and let your users to be just users with no huge time commitment to improving your software.

Solution 3: Be transparent

Here you can earn the trust of your users by sharing the information you have. Make it easy for users to find known issues in a human readable form (i.e. Bugzilla &c do not count). Make sure the people who report issues will know their report has been received and make sure they can follow up to that in case further details emerge.

Also let bug reporters know about unclarities in their reports or, in case you do not intend to change the reported behaviour, just tell them. It’s always better to know what not to expect than to be disappointed release after release (Just imagine Apple had publicly stated that they intended the Finder to remain a mess when releasing Mac OS  X.1, nobody would have been surprised since and there might even be a good replacement by now).

Solution 4: Make reporting bugs easy

This cannot be reiterated frequently enough: Most bug reporting processes are a royal pain. Instead they should be a pleasure. Frequently seen failures are:

Many of these problems come from the fact that bug reporting software is apparently a genre which isn’t designed for usablity or convenience. And many developers - people who may actually appreciate receiving nice reports - don’t seem to see this as a problem. If you think people should include some information about their system in bug reports, give them a one-click menu item to create an e-mail with that information. If you need crash reports or screenshots, tell people about it and prepare as much of the information as you can without user intervention.

Expecting users to manually collect all the relevant information which a machine can easily generate for them automatically while also expecting them to send convenient and well formed reports seems wrong to me. Completely wrong. Cognitive dissonance.

Of course many of these issues merit further and more detailed discussion: Which information can be sent? how should the users’ right for privacy be handled (stuff like X.5 hang logs or system profiles seem to contain a load of information which leaves you with a bad feeling; your bug reporter has to make clear who can read the cases mentioned there along with the log messages or other data users upload)? Who will write the ‘Sparkle for bug reports’ framework? And who’ll lobby Apple into making capturing films of the screen (something I consider a huge time saver for bug reports because you can demonstrate the problem) easy - complete with a reflection of mouse movements, clicks and keypresses?

Who’s reading it

Another qeustion is who reads the bug reports. The larger the software company, the lesser their competence at getting the right people to see those reports. Be it because they don’t read them at all or because the reports are first inspected in some way by support staff who don’t necessarily know what they are doing. Occasionally one gets the impression that the people reading a report have no clue. Such formalised systems also have the disadvantage that you may actually have a better understanding of the problem and of troubleshooting techniques than the person processing it. Which is exactly the frustrating telephone hotline situation you alienate people with. This may be non-trivial for a greedy company, but getting qualified staff in those positions will help.

I’ll finish with a few examples of how bugs are handled by Mac developers. Perhaps people can find a good idea or two in there and steal it.

Apple

The corporate monolith in the Mac universe. There are many ways of getting bugs towards them. The operating system itself offers to send crash, hang and kernel panic logs straight to Apple. As that is a no-effort proposision, it is probably a good idea to do that (at least for crashes, those hang log files seems to contain a load of information that goes far beyond the application which is hanging; it’s also unclear to which extent sending in crash logs for non-Apple applications will make any difference - quite certainly the developer who should see it will never even learn about it). The general guess seems to be that those logs - just as the well hidden but in some sense public bug reporter on Apple’s web site are mainly a source of statistical entertainment for the company. Putting any effort into such reports is quite likely a waste of time.

Apple have other bug reporting systems as well - for example with their developer site - and at least the serious reports like kernel panics seem to be looked at there. Less ‘serious’ issues are a mixed bag. Some are ignored for years, a few are fixed, others are classified ‘behaves as specified’ while most end up being duplicates. With those duplicates obviously marking another unattractiveness of closed bug reporting systems as they mean that more than one person wasted their time pointing out the problem.

The Mac universe seems to think that filing duplicate reports of the same bug somehow improves the situation. I consider that to be wrong. Even if filing the same thing many times gave better results, it’d ultimately be a waste of time, both for the users and for the people processing the reports. It’d also hint that Apple’s bug processing is very poorly managed and that any e-mail from them calling you a ‘valued’ customer is a farce.

Karelia

Sandvox Icon Karelia put quite a bit of effort into their bug reporting system by now. Sandvox comes with its own feedback reporter window which lets you do the things you typically need to do:

Sandvox feedback reporter window

To top things off it also doesn’t fail and die when you don’t have a network connection at the moment you use it, but it saves the report and sends it the next time that’s possible. In addition to that Sandvox automatically discovers crash logs when the application launches and offers to send them in.

More trusting users can turn on automatic submissions of such reports - as well as of exception reports which are only sent once in case the same exception appears several times - which, particularly when using betas, can relax the situation significantly as you know the application will file whichever information is needed without any effort of yours. And if you want to track things or submit additional information you’ll find the relevant case number in your console log.

This system still has a few rough edges but it’s certainly a very good step towards making filing problem reports as painless as possible. In addition to this, you can still submit reports to Karelia via e-mail as they seem to be using FogBugz which appears to provide a reasonable e-mail interface (we better don’t start talking about their table infested web pages which quickly reassure you of their Microsoft DNA).

Panic

Transmit icon The beauty of Panic’s problem reporting system is that it’s completely informal and works via e-mail. That’s step 1 and it is very convenient. The second - and I believe equally crucial - step is that their bug reports are processed by intelligent human beings like Les who do a great job at pulling whatever formal strings need to be pulled in the background and keeping the reporter informed. All this is done completely informally without any numbers or other red tape attached which gives it a very good ‘feel’ even though it may not completely satisfy your inner control freak.

Personally I rather like that style of bug processing because I really don’t want to track the problem I submitted. I observe the problem and if it annoys me enough, I’ll submit it. Then my job is done and the recipients have to figure out what I mean, whether they want to fix it and when they are going to do that. Only afterwards I need to learn about the outcome of this as I can’t influence the rest anyway. And if one can trust the developers to be reasonably good at not losing or ignoring cases just because they didn’t give you a tracking number for them, that’s probably even better than giving you a tracking number.

For beta testing Panic also tried out more formal bug tracking system (Redmine). It is web-based, open source and thus - unsurprisingly - inconvenient and sucky.

Lemkesoft

GraphicConverter icon Many people sneer at GraphicConverter because it never quite made it to OS X UI-wise. That’s a justified critique, but at the end of the day GraphicConverter gets the job done and - unlike many younger and hipper applications - still supports technologies like Mac OS 8 and 68K processors. So I am tempted to think that this is more about having a different focus than about a lack of effort.

In fact, Lemkesoft’s bug report and support system is all about making an effort. For many small problems which I reported over the years - using the e-mail link provided in the application -, I frequently received replies, requests for further details or even beta versions for testing the fix within short time spans; Sometimes within hours. All without case numbers or formalities.

I suppose this way of working doesn’t really scale to larger companies, but it’s still the best experience you can get.

September 17, 2008, 9:17

Tagged as apple, bug, graphicconverter, karelia, lemkesoft, panic, software.

Comments

Comment by LKM: User icon

You’re sending at most a few bug reports each week. Developers receive probably dozens or hundreds of bug reports each week. Spending a few minutes writing a usable bug report is therefore a good investment. If you don’t want to write up your bug in a way that lets the developer reproduce and fix it, it’s better to not send a bug report at all.

September 18, 2008, 16:29

Comment by ssp: User icon

@LKM:
If you have sent detailed bug reports that go beyond writing ‘The Finder is a POS’, you will know that those ‘few minutes’ which developers always claim such a bug report to take are a convenient illusion. That very claim is form of user hostility in itself if you wish.

If developers receive hundreds of bug reports a week and intend to scrutinise them as well as act on them, surely the tools I suggest for making the submission of a fully fledged report easy will pay for themselves rather quickly. If, however, the developer pays only lip service to taking such reports seriously, it will certainly be beneficial to make submitting reports hard and put the blame for their style on the user.

Also, the developers receiving hundreds of reports and not making the process as convenient as possible for their users send a clear statement about how highly they value their users’ time.

September 18, 2008, 17:59

Comment by LKM: User icon

If you have sent detailed bug reports that go beyond writing ‘The Finder is a POS’, you will know that those ‘few minutes’ which developers always claim such a bug report to take are a convenient illusion. That very claim is form of user hostility in itself if you wish.

Being on both sides of this, I’m pretty sure that writing a reasonably good bug report will typically not require more than a few minutes, but will save the developer hours. Obviously, if devs ask users to write good bug reports, having good mechanisms for sending in and keeping track of these bug reports is a must.

September 19, 2008, 23:51

Comment by ssp: User icon

Being on both sides of this, I’m pretty sure that writing a reasonably good bug report will typically not require more than a few minutes, but will save the developer hours.

Perhaps that’s true for simple applications and general setups – i.e. mostly obvious bugs which the developers should be aware of if they used their software themselves. Sure, there are loads of such problems as well and many times a screenshot and a sentence should be enough to indicate the problem.

Slightly more complex software, however, may depend on gazillions of settings which you, as a user, may not even be aware about. The system’s graphics chips may play a role, your network connections may play a role, localisations may play a role and so on (time zones, keychain state, attached file systems, virtual memory state …). To get a good idea about reproducibility of a problem which isn’t totally obvious you probably have to reproduce it in a clean account to get an idea whether any of your personal settings plays a role. And even that doesn’t necessarily make sure that other people can reproduce the problem (because they use different hardware, languages, network setup). Even just switching to a Guest account takes a few minutes thanks to the swapping hell that is OS X. And what if the problem doesn’t immediately reproduce there? Yeah, crap!

Other problems are totally hard to understand for a user and while they happen quite regularly, they can be inpredictable. Examples I have to battle with every day is the fact that Mac OS X.5 (and X.4) usually lose their connection to our wireless network. It remains completely unpredictable but I saw the problem on three different Macs with two different routers while all other machines worked just fine on the networks. It’s not even clear what to report for this problem that significantly reduces the features of my machine. Yet the OS developer doesn’t give a shit, and doesn’t even say what to look out for.

Other ‘tricky’ problems, are potential sluggishness of OS X.5 while Time Machine is running or the habit of QuickTime to jump backwards during playback when there’s heavy disk activity (this may have finally become better since the latest OS or QT update – but I can’t really tell as there never has been a sure way to reproduce the problem). Or look at totally widespread problems like Mac OS X losing the password for its wireless connection. I’m sure hundreds of people wasted their time reporting those over the years and yet it wasn’t terribly effective.

Obviously such issues are worst with the OS as it is the most complex piece of software you are running but other code can be fairly complex as well. As soon as a problem isn’t reproducible on the spot, what would a ‘good’ report do? When Transmit does quirky stuff after it has been running for a week, what can I do except tell the developers about it with no further idea what triggers the problem? And what can the developers to other than shrug and say they can’t reproduce it?

There’s terribly much state in an OS and in modern applications. An – taking a Knuthian point of view – there’s a huge load of magic in frameworks and even runtime environments which even developers most likely don’t fully understand. Capturing all that in a way that makes a helpful bug report seems impossible and – apart from privacy issues – not really the job of a user.

Obviously, if devs ask users to write good bug reports, having good mechanisms for sending in and keeping track of these bug reports is a must.

And that’s the main thing I suggest in my ‘Solutions’.

September 20, 2008, 12:27

Comment by LKM: User icon

Perhaps that’s true for simple applications and general setups – i.e. mostly obvious bugs which the developers should be aware of if they used their software themselves.

The main application I get bug reports for is hugely complex, running on dozens of different architectures, serving hundreds of users, running hundreds of concurrent threads. I obviously don’t expect users to debug the application themselves if they find an issue; what I do expect them is to clearly describe the issue. That includes:

  1. Clearly describe what you did
  2. Clearly describe what happened
  3. Clearly describe what you expected to happen
  4. Include a screenshot (or even better: a short movie showing what you did and what happened) if applicable to your issue (i.e. if it is something “visible”). Do not wrap the screenshot in a Word file if you can avoid it (I have no idea why Windows users insist on sending Word files containing screenshots, but for some strange reason, they often do)
  5. Include system configuration and a list of running threads (we provide a simple tool inside the application which will produce such a comprehensive list with one single click, similar to Apple’s built-in bug reporting tool)

I do not expect users to reproduce the issue on clean systems, or to find ways to reproduce the issue each time if it is intermittent, or anything like this. What is useless to me are bug reports like “Printing doesn’t work”.

I don’t think Brent Simmons or Steven Frank are trying to get people to debug their apps for them. They simply want to teach people who to send in bug reports which are not completely useless.

September 21, 2008, 9:27

Comment by ssp: User icon

@LKM:
It seems we don’t really disagree on things here. I just wanted to stress that it’s important for developers to make submitting the relevant information easy and to automate as much of the technical side of this as possible.

Currently the steps done in that direction exist but are feeble in many cases. That situation may be improving, but in most cases it’s still far from an ideal situation.

For example system profiles are a point on this: On the one hand they can be handy for debugging things. But on the other hand they are a privacy issue. As they contain a huge amount of data, it’s out of question that I scrutinise them and in case I have pre-release (or otherwise NDAed) software on my machine I may not be allowed to share that data with you even if I trusted you on the privacy issues. In addition on the Mac, creation of a system profile alone consumes the proverbial ‘few minutes’ that sending a bug report should take.

September 21, 2008, 16:42

Add your comment

« Symmetries ScriptingMainBlitzschutz »

Comments on

Photos

Categories

Me

This page

Out & About

pinboard Links

♪♬♪

Received data seems to be invalid. The wanted file does probably not exist or the guys at last.fm changed something.

People

Ego-Linking