Blog : Research Studies

Avoiding Agile Disaster

Agile development can be a wonderful thing. Unlike a waterfall approach that can be mired with checkpoints, bottlenecks, and other friction, Agile can free organizations to move quickly. However, with that freedom come deleterious consequences. Chief among them is the loss of  product identity, which leads to an unrecognizable agglomeration of disjointed featuresA blob of garbled parts.

A Blob of Garbled Parts

One of the first questions I ask usability study participants is, “What do you think this thing does?” All too often, the answer is simply “I have no idea.” In other cases participants grasp at random guesses. In the case of Agile development, the cause usually lies with a loss of strategic vision.

Agile works in small, fast sprints that focus on features. In this high-paced product development framework, a myopic mindset often takes hold causing the team to lose sight of the big picture. Rather than asking how each new feature will support the overall product strategy and how each feature will work together to form a whole, teams are just focused on the feature-du-jour. The result is a mishmash of disconnected features–an amorphous blob, not a product. When you ask people what they think it is, you are really giving them a Rorschach test.

This is a problem for an obvious reason. No one wants an undecipherable blob of garbled stuff.

How to Spot the Blob

Luckily, it’s pretty easy to identify if your product is an amalgamation of disjointed features.

  • Open yourself to critically examining where you are.
  • Find some people that have never seen or heard of your product, show it to them briefly and ask them what they think it is and what it does.
  • Allow your subjects to interacted with your product for a few minutes and ask them again.
  • If more than half the people you interviewed cannot tell what your product is or does, you have a blob of disconnected features.

How to Fix Your Blob

This is the difficult part. Often, you have devoted so much time, effort and money into getting to where you are, that it is next to impossible to let go and clean up. Here is what to do:

  • Understand that if you do not consolidate your mess of features into a coherent product, it will only get worse and you will lose more time and money.
  • Without looking at what you have, state your product vision. (E.g. a community for people to share documents.)
  • Itemize all of your product’s features and ask whether they support your product vision. (Do you really need a video editing feature in your document sharing website?)
  • Cut all those features that do not support your product vision.
  • Look at the remaining features and ask how they fit together to form a unified product. (E.g. How does sharing by email relate to new user registration?)
  • If you identify features that do not work well with others, figure out a way to better integrate them.
  • Test the final product to make sure that you actually do have a product that people can understand and want.

How to Avoid the Blob

An ounce of prevention is worth a ton of cure in this case. It is substantially easier and cheaper to avoid losing the products identity than trying to recover it. Below are the steps to make sure that you build a product with a strong identity.

  • State your product vision if you haven’t already done so. (See above.)
  • With every new feature in the pipeline ask how it will (a.) support the product vision, and (b.) fit within the existing whole.
  • Develop a strategy for each feature to support the overall product strategy and to work seamlessly with the other features.
  • Ensure that the design and implementation of each features meets the above two criteria.

The Infinite Pivot and the Death Spiral

We all know them: start-ups that are caught in a cycle of infinite pivots. (I’m sure you’ve already seen the lampoon Vooza.) Sometimes it’s very obvious that a company is pivoting endlessly; other times it is much more subtle. Agile development is very prone to this chronic condition since it is so easy to change tack. What are the tailtell signs that your organization is stuck in an infinite pivot?

  • Your customers don’t know what your product is or what it does.
  • Every new customer support email prompts a new feature or revision.
  • You are often undoing previous work.

If any of the above sounds familiar, your organization might be stuck in an infinite pivot. Of course, pivoting is a vital step in any new company, but doing it too often will erode your product’s identity and leave you with a blob of disconnected parts as well as a fleeting customer base. When things get bad enough, your product can go into a death spiral.

The Infinite Pivot is really just a special case of the Blob of Garbled Parts, although it arises for slightly different reasons. The main culprit in this case is also a lack of product vision but also an over-sensitivity to customer and stakeholder opinion. What I mean by the latter is that the product heads make new product decisions every time they get a new piece of feedback from a customer or stakeholder. Take, for example, a shopping web site. A few customers write in wanting bigger product images, so the product team updates the web site with bigger images in one sprint. Then an investor insists on making the images smaller to fit more products on the screen, so the images are shrunk in the subsequent sprint. Sound familiar?

A strong product vision would curtail this scenario. Conflicting feature requests would be evaluated against the overall product vision. Do bigger or smaller product images support the product strategy? This is dictated by what kind of online store you are building, which is driven by business strategy.

How to Avoid the Infinite Pivot

As in the case of the Blob of Garbled Parts, the emphasis is on clearly stating a product vision and building a product around it. However, it is also important to develop an effective system for incorporating feedback.

  • Customer insights and stakeholder opinions should be viewed as a whole not piece by piece. For example, how many customers complain about the product images? Do more people want smaller images or bigger ones?
  • Each feature request should be scrutinized to see if it fits with the overall product vision as well as with existing parts.
  • If the feature request makes the first cut, one must guage its feasibility and its priority vis-a-vis other features in the pipeline.

Following these steps should help to ensure that you do not change tack too frequently and maintain a strong product identity.

Stay True to Your Product Vision

In my experience, the most common danger associated with developing products in an Agile framework is focusing on building individual features rather than a product. By clearly defining a product vision and ensuring that all development supports that vision, you can focus on building something that your customers will understand and, more importantly, want.

People Prefer Choice over Better User Experience

Recent research suggests that if consumers perceive that their freedom of choice is limited, they will often switch to a new product from one with which they are already familiar,  (“Why Dominant Companies Are Vulnerable“,  MIT Sloan Management Review,Winter 2012). The researchers, Kyle B. Murray and Gerald Häubl, explain that this phenomenon might be one important reason why market leaders such as Microsoft lose dominant market share over time. For example, consumers might opt to switch to the Firefox web browser and endure the cost of learning a new software simply to exercise their freedom of choice. Not only that, Murray and Häubl found that consumers might make the switch to the competitor even though the competing product is not as good.

The experiment consisted of websites with different interfaces that allowed users to search for new stories. Some participants were allowed to choose the website to use while others were not. Specifically Murray and Häubl found:

51% of consumers who had no choice in selecting the interface they learned to use switched to a competing website as soon as it was available. By contrast, among consumers who were free to choose the website they would learn to use, only 23% switched to the competitor, despite the fact that other users rated the competitor’s website superior on several dimensions (including ease of use, fun, efficiency and effectiveness)… [We] found that the market leader’s advantage in being able to install a set of nontransferable user skills in its customer base is offset by psychological reactance, a force that motivates people to act against perceived constraints on their freedom of choice.

Murray and Häubl go on to explain:

As people learn to use a particular electronic interface associated with information search or online shopping, for example, they often become locked in and develop extremely high levels of loyalty even when otherwise equivalent competitors are available; the cost of switching outweighs the benefit of using another product. However, our research indicates that the depth of loyalty weakens when consumers feel that their freedom to choose is restricted. Specifically, as people feel that their choice is constrained and that one interface dominates the market, they react against the constraint by turning away from the market leader’s offering, thereby subjecting themselves to the associated costs of switching.

What does this mean for product strategy? Strong-arming customers to stick with a particular product might actually alienate them rather than foster their loyalty.

Market Research and the Primitive Urges of the Consumer

“The trouble with market research is that people don’t think how they feel, they don’t say what they think and they don’t do what they say.”

ThirdSight Software on a Smartphone Decoding Expression

The BBC reports on an upcoming breakthrough for market research, currently being developed. Dr Roberto Valenti of the University of Amsterdam and Dr Theo Gevers.

The two have established a company, ThirdSight, to take advantage of computerized emotion recognition (decoding emotions from facial expressions). ThirdSight has successfully run its software on a smartphone, but the team acknowledges that results are not yet perfect, requiring a researcher to oversee the software, because it cannot decode context or hidden meanings. For instance, it considers both a happy smile and a bewildered smile as ‘positive’.

This technology poses some promising power in the future of market research.

Read full BBC article »

The Real Life Social Network

I loved this presentation by Paul Adams of the Google UX team. He explores designing for real social networks by examining relationships, influence, identity and privacy.

The entire presentation is extremely well done, and the discussion around relationships and our online versus offline social network truly illuminates important factors in social design.

Read More

Ratings by Communities Are Skewed—Now What?

Many online and mobile applications rely on ratings and reviews from their communities to provide wisdom for their remaining users. Services such as Yelp, Amazon, Digg, and even the Apple App Store use input from their users to evaluate some intrinsic value of a set of items—be they books or iPhone applications.  However, new research recently published in the MIT Technology Review suggests that the wisdom of crowds can be inaccurate and misleading. Does this cast doubt on the utility of community-driven rating systems?

Vassilis Kostakos, an adjunct assistant professor at Carnegie Mellon University and his team confirmed that the rating systems commonly used can “easily be swayed by a small group of highly active users.” The Technology Review article goes on to write that “rating systems can tap into the ‘wisdom of the crowd’ to offer useful insights, but they can also paint a distorted picture of a product if a small number of users do most of the voting.”

Although Professor Kostakos’ research validates a suspicion that many have had, it does not necessarily mean that community-based review systems are useless. The article states:

Jahna Otterbacher, an assistant professor at Illinois Institute of Technology who studies online rating systems, says that previous research has hinted that rating systems can be skewed by factors such as the age of a review. But she notes that some sites, including Amazon, already incorporate mechanisms designed to control the quality of ratings–for example, allowing users to vote on the helpfulness of other users’ reviews.

Kostakos proposes further ways to make recommendations more reliable. He suggests making it easier to vote, in order to encourage more users to join in.

What this means for the design of interactive products with such rating features is that steps should be taken to ensure a more representative outcome of user-driven reviews. The following factors can be considered to that end:

  • Count only one vote per user.
  • Provide a mechanism for users to vote on the usefulness of written reviews, and factor that into the total score.
  • Make it easier for all users to vote to capture a broader cohort.
  • Factor in the network patterns of user voting. For example, if a group of users consistently votes together on items, perhaps compensate in the algorithm for that behavior as it tends to skew results.

Eye-Tracking Studies at Google

Two user experience researchers share on the Google Blog how their team conducted eye-tracking studies on the interface of Universal Search to gain insight into optimal information design. They write in their post:

Our User Experience Research team has found that people evaluate the search results page so quickly that they make most of their decisions unconsciously…. Of course, eye-tracking does not really tell us what they are thinking, but it gives us a good idea of which parts of the page they are thinking about.

Read More

Modern Mobile Phones Frustrate Most Users

The BBC reports on a study conducted by Mformation, which reveals that of 4,000 people interviewed in the UK and US, 61% claim that “setting up a new handset is as challenging as moving bank accounts.”

The report reveals other details of the complexity users face, such as using various applications, browsing the web, reading email, and sending picture messages. Results include:

“Of those questioned, 95% said they would be more likely to use new features if the initial set-up were easier.”

“Some 61% of those questioned said they stopped using an application if they could not get it working straight away.”

“Having icons for all a phone’s available services at hand was better than burying them in a sub-menu …”

via: Experientia

On Usability Problems with Voting Machines

Today is the big day, and no matter for whom or what you are voting on November 4th, you not only want your vote counted, but you also want it counted correctly. In the spirit of fair elections with a twist of usability geekiness, we at Montparnas compiled a few resources where you can learn more about usability of voting machines.

Usability in Civic Life: Voting and Usability Project

The Usability¬† Professionals’ Association (UPA) has been running a great project that seeks to evangelize good usability in voting machines. It’s one thing when it’s difficult for a user to add an item to a shopping cart, but it’s a whole different ballgame when votes that determine a presidential election are miscounted or not counted at all. Usability in voting machines is perhaps the most important application of the usability engineering field. The UPA writes on their site

Read More

The Kiosk Experience

As a designer and consumer advocate, I often judge the experiences that I have with various products and services. So I was keen to read David Pogue’s recent article with his own experience observations, aptly titled It’s the Software, Not You.

Of the Delta Airlines touchscreen kiosks, Pogue writes:

“Whenever I encounter badly designed software like this, I stand there, slack-jawed, mind boggling, and wonder what on earth the designers were *thinking.* Not, obviously, about elegance, intelligence and simplicity”

Beyond kiosks, Pogue also mentions the PalmPilot (as a good example) and touchscreen payment systems in taxis. The article got me thinking about the various examples of kiosks, good and bad. I recall going to the movie theaters about two years ago and being pleasantly surprised that all I had to do was insert my credit card, and there my tickets were… tadah! That simple.

It’s always great to be pleasantly surprised by the devices you interact with, but it’s not a simple thing to design them. It takes a lot of thought and research to minimize the steps and customize to the user’s needs. Most importantly, knowing what the most common tasks are can be invaluable, particularly for kiosks which are meant to speedily get people through common tasks. Holger Struppek of Hot Studio writes in-depth about one such design exercise for Wells Fargo’s ATM’s.

I also recommend reading the research reports on kiosks from Witchita State’s SURL (Software Usability Research Laboratory): Designing a Touch Screen Kiosk for Older Adults: A Case Study and How Important is Visual Feedback When Using a Touch Screen?