Two Tips For Usability Testing On Mobile Devices

Posted on June 13, 2011 by Ryan Schroeder

Last week we ran a usability study of a recently released mobile website. Since testing on a mobile device is a bit more difficult than on the desktop, I wanted to share a couple of details about our setup and what we learned.

Avoid complicated camera rigs.
There are lots of great tools for recording a usability session on the desktop, but most aren’t designed for testing on a mobile device. So how to reliably capture a session? You may have seen examples of people attaching a camera to a phone in order to record a user’s taps. We evaluated a few similar options, but they all seemed too intrusive. These setups constantly reminded users they were being recorded. In addition, most added a lot of extra weight to the device making it very difficult for a participant to use the device naturally. 


We found that mounting a high resolution HD camera on a tripod high above the participant’s shoulder allowed us to adequately record a session. For some users, it took a bit of panning and zooming to follow their hands as they used the device, but most required very little adjustment of the camera. And with a standard zoom, we were able to get a surprising amount of detail. This allowed us to stay out of the way — after a few minutes, most users forgot they were being recorded.

Let participants use their own device.
Avoiding a complicated camera rig had another advantage — it allowed us to let participants use their own devices. We found allowing people to use their own phones really helped make them feel more at ease. However, with people bringing in their own devices, we weren’t able to set anything up beforehand. So, in order to quickly get started and to avoid asking participants to type in large URLs on a tiny keyboard, we created short URLs for the website being tested and texted them to users. We asked permission first, but everybody was more than comfortable with the idea. Of course, it helped that we were testing a publicly available, production website.

Making participants as comfortable as possible.
Any usability session can be awkward for participants, and a mobile device study is no different. Participants are often required to sit in a sterile room with a one way mirror while a camera is pointed at them and someone is asking them dozens of questions. We found that both of the tips described above helped to ensure people were comfortable. The more comfortable people are, the more natural they’ll act — and the better feedback they’ll provide.

Posted in Usability Testing | Leave a comment

User Testing: High Touch or No Touch? (Part 2)

Posted on May 12, 2011 by Mark Wieman

As I mentioned in my previous post, we do primarily two kinds of usability testing here at Fell Swoop:

  • High Touch – This is the traditional method of usability testing. It involves sitting down with the user and talking with them as they perform a set of tasks.
  • No Touch – This method is a little different. It involves giving the user a set of tasks to work through on their own, then watching a video of it.

While “high touch” often uncovers deeper insights, it also takes longer and costs more. The “no touch” approach can yield results in a matter of hours instead of days. It’s also cheaper because it doesn’t require a physical space, and participants are paid less for shorter sessions.

Intrigued? Let’s take a look at the process for conducting a typical “no touch” user test.

  • Step 1: We create an online prototype of the design to be tested.
    This prototype can be as simple as a series of PNGs, a clickable PDF, or a basic HTML website. We post the prototype at a private URL and take it down after the test is over.
  • Step 2: We recruit three to five participants based on a handful of basic parameters.
    Age, income, education, and level of technical savvy are all things we can define. Getting more specific is possible but requires more time. The beauty of “no touch” is its speed, so a more general user profile is typically preferred.
  • Step 3: We create three to five tasks for participants to complete while using the online prototype.
    During the sessions, these tasks are displayed in the context of the prototype itself. Participants see each task above the section or page to which it refers.
  • Step 4: Participants record themselves completing the tasks using the prototype.
    Sessions last only about 20 minutes and don’t involve a facilitator. Participants do the sessions wherever they want using their own computer or mobile device. Typically, the sessions are completed on the same day, sometimes mere minutes after we post the prototype and tasks.
  • Step 5: We review recordings of the sessions and put together a simple list of findings.
    We analyze what participants say and do while using the prototype. Obviously, we can’t probe the users for additional insights, but we can always run a follow up test with different participants to test new issues or opportunities.

The best thing about the “no touch” approach is it allows us to test early and often since the whole process can be completed within a day. This means more user feedback can be incorporated into a design, making it that much easier to use.

The traditional “high touch” method of user testing can’t be forgotten, however, because it results in the deepest insights. Therefore, a perfect user centered design project would likely include a mix of both “no touch” and “high touch” user testing. Multiple rounds of “no touch” and one or two rounds of “high touch” would be ideal. This is uncommon now, but we bet it will become the norm soon.

Posted in Uncategorized | Leave a comment

User Testing: High Touch or No Touch?

Posted on March 28, 2011 by Mark Wieman

There are many flavors of usability testing. At Fell Swoop, we have two favorites:

  • High Touch – This is the traditional method of usability testing. It involves sitting down with the user and talking with them as they perform a set of tasks.
  • No Touch – This method is a little different. It involves giving the user a set of tasks to work through on their own, then watching a video of it.

Each method has its advantages and disadvantages. One of the best things about “high touch” has to do with uncovering motivations. When the user does something interesting, we can ask them why they did it. With “no touch,” motivations can remain hidden unless users are good at thinking aloud.

Below is an example of how “high touch” usability testing can pay off in the form of user behavior insights.

I once redesigned a web app for a car sharing company. The app allowed customers to reserve cars. It essentially asked them when they wanted to have the car, then showed them which cars were available. During user testing, we noticed an interesting behavior. Users wanted to know which cars were available for their trip, but they often picked cars that were available after their trip. Probing on this behavior revealed the reason. Customers had a strong aversion to inconveniencing other customers. They wanted to choose times least likely to inconvenience others if they had to return the car late.

After learning this, we optimized the design to make it easier to see availability after the requested trip times. Had we gone with a “no touch” approach, we may not have known why users did what they did. This is the beauty of the “high touch” approach. Despite being more time consuming and costly, it can result in better designs and happier customers.

In my next post I’ll describe our experience with the “no touch” method of usability testing and tell you why we think it’s perfect for some situations.

Posted in Usability Testing | Leave a comment

In-Flight Ethics, Etiquette, and Customer Experience

Posted on March 21, 2011 by Matt Dente

On Thursday, The Wall Street Journal published the article “So Who Gets the Armrest?” a light piece that reporter Scott McCartney summarized as the “Ethics and Etiquette To Survive Cramped Seats, Crying Kids, and Stinky Food at 30,000 feet.” At its core, this is a human interest story that ponders questions such as “You’re in the middle seat, between two strangers. Who gets the armrests?”

The answers are revealing, especially former Continental Airlines CEO Gordon Bethune’s response to the question “Do you recline your seat?” to which he answers “Live with it. The recline is your space.” Meaning, tough for the person preparing that big slide show presentation on a laptop behind you.

So who get's the arm rest? Wall Street Journal

Illustrations by Jonathan Carlson, The Wall Street Journal.

There are other insightful thoughts from veteran flight attendants, frequent flyers, and even an ethics professor, all well aware that our generally well behaved populace becomes rude and uncharacteristically mean when sharing cramped space during a long flight.

What surprises me in this piece is not that passengers can become inconsiderate to one another under the right conditions, or that people complain too much, but that an airline hasn’t differentiated themselves by using these facts to their advantage. The responses from industry insiders in the article imply that the onus is on the passengers to sort it out amongst themselves, but what if the airlines got involved? Could they improve the customer experience and build more customer loyalty without throwing mounds of cash at the problem?

I think they can, and not in the most expected of ways. Perhaps the airlines can address this problem without redesigning their passenger seat layouts (costly), and instead by redesigning how they deliver information, set expectations, and offer incentives (not as costly).

Inform and Educate

In the United States, where I live and travel most frequently, the cultural norms regarding basic manners and consideration for others varies quite a bit. While it’s not the job of the airlines to educate passengers on proper manners, they could provide some ground rules that make traveling better for everyone. The best place to start is by informing and educating during the typical cabin safety demonstration. Reminding passengers that they are sharing the cabin space and are “in it together” for the next few hours could go a long way towards improving the customer experience without spending a lot of cash. Including this information during the cabin safety presentation, inside airline magazines, and on signage behind each seat could help. Imagine a small sign in front of you reading, “don’t forget to return your seat to the upright position during meal service so the person behind you doesn’t spill their hot coffee all over themselves.”

Design a Passenger Based Incentive System

Incentive programs could also be established to reward considerate passengers. Imagine completing a short survey at the end of each flight asking you to rate the consideration of those around you. Was the person seated immediately in-front of you, A) Very considerate, B) Somewhat considerate, C) Not considerate, D) Playing video games with the volume turned to 10 the entire flight. Joking aside, positive scores could lead to points that can be redeemed in miles programs (which are costly), but also in softer benefits like early boarding privileges, or reserved seating in prime cabin locations reserved for highly-considerate passengers.

While these programs would cost the airlines something, it would be cheaper than removing rows in the planes to create more passenger space, and it would address the problem from a different angle, one that would theoretically improve the experience for passengers.

How would people feel about these types of programs? Would passengers balk at airlines attempting to control our behavior? Maybe, but airlines already limit behavior for our safety. We are instructed not to unbuckle our seat belts and stand up before the plane is parked at the gate, or not to use our iPods during take off, and we listen. Maybe airlines should start suggesting behaviors that improve customer experiences, and then maybe they will see a little more customer loyalty and an improved brand perception, and ultimately, a little less complaining.

Posted in Business, Customer Experience | Leave a comment

Conducting a Customer Experience Audit: Beyond the Web

Posted on March 13, 2011 by Matt Dente

At Fell Swoop, our primary focus is in designing the user experience of websites, applications, and interactive products, but user interface design is really only one facet of what we are considering on a daily basis. In a recent audit conducted for Clear.com, we analyzed the end-to-end experience of a prospective customer shopping on the website, the unboxing experience after they’ve received the product, and the continued use of the account and support websites post-purchase. When designing an ecommerce website, the shopping, configuration, and checkout processes are obviously critical, but the unboxing experience and the account setup and bill paying experience are equally important. All of these touch-points shape the customer’s perception of a company; its brand. Audits such as this help us uncover opportunities to improve the overall experience, on the website and beyond.

Posted in Customer Experience, Video | Leave a comment

Starting a New Business: Lessons Learned

Posted on March 7, 2011 by Matt Dente

When I co-founded Fell Swoop just over two years ago, the world around us was in flux. The economy was tanking, financial scandals were rearing their heads, and corporate spending was drying up. In hindsight, Fall 2008 may not have been the ideal time to start a new consulting business, but I believed in what we were doing and I knew it was time to do it our way.

Two years later, I couldn’t be happier with the progress we’ve made, our talented team, and the fantastic clients we get to work with everyday. While we’ve been profitable from day one, it hasn’t been picture perfect every step of the way. In fact, over the last two years, I’ve learned some invaluable lessons about running a business, serving clients, hiring staff, and designing user experiences. While I’m sure I’m just scratching the surface, and I still have many, many lessons to learn, there are a few topics worth writing about on this blog. Over the coming weeks, I’ll share some of these lessons as well as some practical common sense knowledge gained since starting Fell Swoop in 2008.

Just to be clear, this is not an advice column. I’ll share lessons I’ve learned while running Fell Swoop, and they may apply to you and you may learn from them, but there is no guarantee. In fact, the first lesson I’ll cover in the series is about designing your own business by applying user-centered design principals towards creating a system that works for you and your team.

Why write this series? The answer is two-fold; 1) So the people that choose to work with Fell Swoop know exactly who they are dealing with, and 2) so we can help others in our industry benefit from Fell Swoop’s experiences. That’s it.

Here is a peek at the topics on the list:

  • Lesson 1: Designing Your Own Business
  • Lesson 2: Being Flexible
  • Lesson 3: Admitting When You Are Wrong
  • Lesson 4: Being a Partner, Not a Vendor
  • Lesson 5: Choosing Fixed-Fee vs. Time & Materials

I’m certain that I’ll learn more about these topics in the coming years, so don’t be surprised if I’m singing a different tune at some point, but for now, I can’t emphasize how important these lessons are in driving our business forward.

Posted in Business | Tagged | Leave a comment