Thoughts on A/B Testing and Wireframing Tools

Posted in Design.

During my recent visit to Chicago for the Cramer Dev quarterly get-together, we attended the Windy City Rails conference. While I’m anything but a RoR developer, I still learned some interesting stuff about the language, as well as some general principles that can be applied to font-end development. Ryan Singer of 37Signals, gave a great talk entitled UI Fundamentals for Programmers. It was a very informative session, and in some ways, it has changed the way I approach the designing process.

Ryan believes that the UI is a really important layer of software because it’s what the users see. I happen to agree with him. In his presentation, he discussed some simple, yet key concepts for designing and implementing UI for applications. More specifically he talked about communicating with the user through on-screen language, as well as using visual techniques to consciously control what the user should be focusing on. He finished his presentation by talking about some techniques for implementing templates and helpers.

At the end of Ryan’s presentation, he took some questions from the audience. There were two questions in particular where I thought he made some rather bold statements about A/B testing and wireframing software. While every designer holds different opinions regarding these topics, I’d like to voice my dissent with Ryan’s views.

A/B Testing

When Ryan was asked for his thoughts on A/B testing, his reply was “We don’t have a lot of patience for that (testing).” I don’t have a lot of patience for A/B testing myself because it’s time consuming and it’s a total drag to create several variations of one function, or an entire web page for that matter. However, making the perfect UI, which would look and function differently for everyone, is an impossible task. I’m not saying that everything needs to be A/B tested on a web app, but to have the attitude, “make one version that makes sense to us and launch it,” means you’re ignoring the user experience for a potentially large number of your users.

Ryan also stated that “testing happens on the marketing side but not within the app.” More power to them. I’m sure 37Signals has crunched the numbers and knows whether it’s worth their time in dollars and cents to do A/B testing within their software. But from a general standpoint, a company could potentially be loosing out on a lot of profit all because they think A/B testing “is a drag.” What I’m saying is, don’t write off A/B testing because you don’t like to do it. Research and figure out if it’s worth your time and resources. In the end, it’s all about the ROI.

Another point I’d like to make about A/B testing is what makes sense to one user will not make sense to another. Design is very subjective and while a design may be aesthetically pleasing based on principle, not everyone is going to like it. Even though we’re inevitably going to alienate some of our users, the goal is to alienate the least amount as possible. If done properly, A/B testing can provide solid data about how your users respond to certain elements of a web site, including design aesthetics. One thing we might learn is that our users’ perception may be vastly different from our own. A very good book that talks about these principals in more detail is Don’t Make Me Think by Steve Krug.

Wireframing Software

Another question that came from the audience was asking Ryan about his thoughts on tools like Balsamiq, or wireframing tools in general. Ryan’s response was “I don’t understand what they are for.” I loathe wireframing software and would much rather work out ideas on a whiteboard or on paper. Ryan does not interact with external clients when designing for 37Signals, but I do. The reason why wireframing software is necessary is because a client expects more than just some doodles on a piece of paper.

Whereas Ryan might dive right into Photoshop or writing HTML, it doesn’t seem cost-effective to me to follow that process. My typical client tells me they want a website and that’s about all the info I get. If I didn’t ask any probing questions about features or content, I don’t think I’d get too much more information from the client until it was too late into the design process. Wireframing helps to breakthrough some of the informational and layout challenges. It would take a lot more time and resources to achieve the same goals once you’re in mock up or development phase of a project.

Everyone has different opinions regarding A/B testing and wireframing software. I’m sure there are plenty of folks out there who would argue my points, and I’m not saying that my methods are the right way. There are multiple paths to achieving a finished website, and it will vary depending on the size of an organization, resources, and personal preferences.