Toggle navigation

Install and Update Usability

One of the nice things about working in a small, flat organization is that you have to do support. With no levels of hierarchy and everyone monitoring support, communication is easy and fixes are fast. My experience today in trying to short circuit what appeared to be a small issue that is causing some pain to customers is a good example of how small can mean nimble and responsive.

The past few days of my first week I've been monitoring support and answering a few Get Satisfaction questions where I could. That's a part of everyone's job here. But it also helped with one of my primary goals this week, which was to get get a handle on the priorities and begin working on a few of these small projects. I've been getting up to speed on the apps, filing bugs, and looking at the UX priorities. I've also been thinking about how to attack the low hanging fruit. What I saw while doing some of this work in parallel, is that I might be able to do some triage and kill a few birds with one stone here and there.

The Upgrade Problem

I saw a few posts about issues people have with upgrading, knowing I had the same discomfort with the upgrade process myself. But I got back to work, and filed the issue in the back of my mind. I've also been doing a content inventory of the small site, and trying to get a handle on the IA there. I was looking at the really long and verbose pages we have, and in the process was going through scenarios like finding the upgrade link. I knew it would be hard for some people to figure out. It was too much work.

Then I came to the AIR upgrade badge and wondered how much of an issue that was. If you can think back to the first time you installed an AIR app, you might recall how seemingly quick the process was, when compared to downloading a package and executing it on your machine (running setup.exe on Windows or dragging a .app from a disk image on Mac). It's almost scary how seamless AIR installs are after that. Almost.

The issue for some with AIR installations, especially upgrades, is that they introduce different way of upgrading than what we're familiar with. You won't know what it feels like until you do it once. Hell, it can feel unfamiliar for a few times until it you've done it enough that it feels natural. I thought the only way to really know if this was pissing people off was to test it.

Doing Ninja-Usability Testing

There are various gradations of usability testing, but the ones I'm interested in are those that are cost effective while being easy enough for me to do quickly and remotely. Discount usability testing is about low cost methods for testing using only a handful of people that run through scenarios. The outcome is often that the most obvious problems are uncovered with the least investment in time and resources. Jakob Nielsen originated this idea. A radical version of this is the 5 second test, which I became aware of after reading an article describing the process at UIE.

For my needs, I wanted to just figure out this one problem. What do people experience when they try to upgrade Mockups for the Desktop? So I turned to remote usability testing tools. OpenHallway is one I've been testing out--they're a sponsor of Konigi and I'm writing a more in-depth of review of the service soon. What I wanted to do was to give people this simple scenario.

You've been using Balsamiq Mockups, and you want to upgrade to the latest version. Can you find the upgrade installer in the site?

Get to the point where you think you've found it and tell us where you think it is. Tell us what you would do at that point to upgrade.

So I put a call out on Twitter and stopped at 6 respondents. I wanted to test 1 problem with a few people. I don't know much about ninjas, but that seems pretty ninja-like to me.

The Feedback

This the feedback I got. My notes are quick and dirty. There are no task markers because there's only one task. There's no quantitative data to analyze. Just observation and common sense.

Tester 1
Wrote: You click on the "Install now" button

Notes: Started by scrolling through home page. Cliked Support > FAQ. Succeeded after searching the FAQ. Succeded in under 1 minute.

Tester 2
Wrote: After navigating through several pages, I found out that to upgrade the software, you simply download the latest copy off of the site.

Notes: Used the Support link after skimming the home page. Scrolled through the Support page and didn't find it and returned to Home/Products link. Clicked Desktop link in hero. Then appeared to read the page until scrolling down to Upgrade instructions. Found it, didn't sound confident at first. about 1'30" to complete task.

Tester 3
Wrote: Final thoughts:
1/ difficult to find download area
2/ difficult to see if by downloading the latest version it will be installed without deleting anything
3/ looks a bit like apple website without the blingbling effects

Notes: Skimmed the home page. Clicked the Products page, found home/products confusing. Went to desktop mockups product page. Then support. After filing with browsing, saying "not very useful stuff," went to search for "download" and tried the Examples page. Found the Desktop link and guessed that it was an install process.

Tester 4
Wrote: it wasn't there or at least i couldn't find it i was looking for upgrade or something similar

Notes: Failed or gave up task.

Tester 5
Notes: Scanned the home page for an upgrade link. Clicked Desktop product link and found the Upgrade heading below the fold. Then wondered "Would download the demo is the same as just downloading? I certainly don't want to buy it again." Successfully started the download, but wondered if she was doing the right thing.

Tester 6
NOTE: She took the test AFTER I had already made changes by creating a download page and adding the download link to the Header.

Wrote: The first thing I was looking for was a download link. It was on the top right of the first screen in the main navigation. I would have expected that if a new version is recently out that the home page would promote it more obviously though (so I looked in the main area before I looked at the navigation, which is where I expect to find it in a permanent fashion).

On the download page I had to interact with the Flash widget in order to get the download started. That was not expected, I was expecting to see some obvious "Download Version X for Y Platform" button - don't really understand the benefit of that little widget.

All in all it was straightforward to do it, but things just did not look like how I expected. The site could really use a visual refresh so that core content, navigation and important call to actions (like "Download the latest X") are visually differentiated.


Notes: She was successful in finding the Install badge quickly, but again the default AIR install badge is not familiar enough.

Taking the Low Hanging Fruit

Did you notice the funny note on the last one? She took the test after I had already made the changes after seeing what needed fixing in the first 5 tests. You could argue that those fixes were common sense anyway, but the audio from the users thinking out loud provided even more feedback about specific things with the AIR badge and what they expected. I didn't transcribe those notes here. Below is the plan I wrote up.

Immediate Plan
1. Create a download page with very little copy. Be sure to put in right keywords in title for search.
2. Add Download link in header.
3. Add a more prominent Download call to action in support page.
4. Think about interim ways to make call to action clear in home page without a major redesign.

Upcoming Plan
1. Vet the design of the updated download button (AIR badge) we already mocked up.
2. Add a link in Balsamiq or Help menus of desktop that goes to our new download page.

I've already done 1 and 2 from the first list. Download is now in the header and links to the new Download page.

Lessons Learned

It's been a fun first test and it was good to try using a service that's simple enough for my needs. I started by putting a session together at around 7am. I recruited via Twitter, got 6 respondents and waited while they ran their tests. By 10 I was watching their videos and wrote up notes and a plan for changes. I had lunch, and was making the changes by 1pm. That's how I want to do things around here.

Mike for the Balsamiq Team

Leave a Comment

Comments (11)

  1. Deena,

    1) We tested with the live site.

    2) Users did record their audio. It was important to me to hear people think out loud, but it’s difficult to ensure that they do with unmoderated tests, obviously. Some were better than others.

    3) No, we didn’t compensate our users.

    The test was quick and simple and addressed a problem that I identified in my first month as needing immediate attention.

  2. Hi Mike,

    This is a nice post, but it would be a lot more helpful if you added a few more details of the logistics of the tests. These are my questions:

    1. Did you test the live site? You mention making quick changes – was that all done on the live site?

    2. Did you get your users to record their audio, or was that not important to you? This is something I’ve specifically struggled with in our Open Hallway tests, so I’d love to hear whether you succeed and how you went about it. Not having audio really limits the value of the tests, unless what you’re testing is uber specific (which may have been the case for you).

    3. Did you compensate your testers? If so, how much?


  3. Thanks for the suggestion Norman. Will pass it on.

  4. Hi Mike,

    Regarding the caching problem, probably the easiest thing to do is include the version number in filename of the update app viz; MockupsForDesktop-1-6-67.air I appreciate that it’s a bit more work for you to make the release available this way.


  5. Thanks everyone, for the feedback.

    @Norman, that is a problem. I’m not sure how to rectify it, however, if you’re not able to see the updated version on our server because of your network’s proxy servers.

  6. Great post Mike! I found it incredibly informative as I’d like to put this sort of testing process in place at our small startup. Loved the Twitter recruiting.

    I look forward to your next post.


  7. Great stuff! Looking forward to hearing more of these kind of down-to-earth experiences in doing real UX work for Balsamiq.

    I have to say I think Balsamiq made a great choice in not only hiring you but giving you the freedom and time to post this stuff—I’ll surely be checking back!

  8. Hi Michael,

    Firstly, welcome aboard, Balsamiq sounds like a fun place to work.

    Secondly, a problem several of us have been having is aggresive caching by our corporate networks, the result of this is that I can click on the upgrade/update widget and it will download something, but, when it then runs it runs the version cached on our servers and thus says ‘this version is already installed’. To upgrade I have to get the upgrade at home & bring it into work on a memory stick.


  9. PS. You also need to work consistency of your header throughout the site, the blog header displays “Company blog” instead of download 😉

  10. Nice move. I can find the upgrade since I have learned the site structure, but it always was weird to me how the download/upgrade/install widget is in the middle of a long page among some other info which at the upgrade time is completely irrelevant.

  11. A quick note: the pre-release of Mockups now has a “Check for Updates…” link which goes to the new Download page. Check it out!