John Hughes, WALL-E, and writing for a real person

I was a teenager at that magical time when John Hughes was doing his best work. I was 14 for “Sixteen Candles,” 15 for “The Breakfast Club,” 16 for “Ferris Bueller’s Day Off” and “Pretty in Pink,” and 17 for “Some Kind of Wonderful.” (Yeah, I’ll wait for you to check IMDB and calculate my age.) I felt like he was writing to me, for me, and about me.

Turns out, he had a secret weapon. Her name was Alison.

After a teenaged Alison’s angst-filled letter to John Hughes was met with a form letter as a response, she angrily called him out on it. He responded – really this time – and the result was two years of correspondence.

Though the letters tapered off, they did have a phone conversation years later:

“…he was glad I had gotten in touch and that he was proud of me for what I was doing with my life. He told me, again, how important my letters had been to him all those years ago, how he often used the argument ‘I’m doing this for Alison’ to justify decisions in meetings.”

This reminded me of a similar story about WALL-E and the girl who cried. It seemed a woman named Courtney would cry every time she watched the WALL-E trailer, so she videoed her reaction and posted it on YouTube. It found its way to the Pixar offices, and was passed around:

“Six months ago, when the first trailer for WALL-E came out, we were only halfway done with the film, and we weren’t exactly sure how we were going to get it done. We were exhausted. And then, one day, a movie showed up on YouTube showing a girl watching the trailer for WALL-E. And every time she watched it, she would cry on cue. When we saw that, we knew we were on the right track.”

The people at Pixar sent her multiple emails during production, and when WALL-E was finally released, flew her in for the wrap party. They even introduced her during the speech before the movie, and she stood up to thunderous applause.

These were real people with real emotions and real opinions engaging with products and those who created them. How much more powerful would it be to say, “I’m writing this for Alison” or, “How would Courtney react” than it would be to say, “What would this made-up persona think of this”? How much more interesting would it be to strike up a conversation with a living human being than it would be to hang stock photos on the wall and pretend you’re having a conversation with them?

Personas are fine. Not suggesting you toss their lovely, modelesque faces in the shredder. But what if you supplement the personas with someone you actually know? Are you more apt to focus and fight for content that will benefit your brother than you would some dude ripped from the archives at iStockPhoto? I’m going to guess you would.

Find the people you’re writing for. Ask for their opinions. Mention their names in meetings. Fight for them.

Ferris would have wanted it that way.

Measuring your content with user data

I wanted to build on Keri’s recent insightful post on measuring content strategy, and talk about possible ways to measure the effectiveness of content changes on e-commerce sites.  More specifically, how do you select the best content if you have a variety of different alternatives, each with its own group of fans who want to get it on the site right away?  Since the voice of a web site can be such an abstract, arbitrary decision, how can we apply methodologically robust research methods to help make these decisions?

First, I would define “effectiveness” in this context as the optimization of the following three concepts:

• Do users understand what you are trying to tell them and what action they should take to be successful in their task?
• Are you invoking the desired emotions with your content?
• Does the proposed content result in higher conversion rates than other alternatives?

It’s so important to combine the user perception data (the first two bullets) with business metrics (the last bullet).  From my experience the only way for user experience professionals to affect change is if we can show the positive impact these changes have on engagement/revenue metrics.

It seems to me that you will be well served by using the following three methodologies to measure the relative effectiveness of different versions of the same content.  This is also a really nice way to progressively reduce the number of alternatives down to the best solution:

• Usability testing. Start with several different version of the content (~10), along with the current version (if it exists).  Ask users in a lab setting what they understand the content to mean, and any other thoughts they have on the way it sounds.  This should help narrow down the alternatives to 4-6 possibilities.
• Desirability testing.  Use the Desirability method, but adjust it for use in large sample online surveys by turning it into a between-subjects experimental design.  In the survey, users are asked to rate the content on different brand and design attributes.  This way you can determine what emotional response the content extracts out of users.  You’d also be able to ask users which version of the content they’d prefer, and why.  This method has the added benefit of large numbers to give you confidence in the statistical significance of the results.
• A/B testing.  Once you’ve narrowed the alternatives down to two or three, live A/B testing can help you determine which of the alternatives perform better from a revenue or engagement perspective, by looking at differences that can be attributed purely to content changes.  This obviously works easiest when the content is directly related to a revenue-generating task, like the call to action on a checkout page, for example.  But it’s not just about revenue — there are great ways to measure metrics of engagement with the page, which is just as powerful.

Now, I can see two issues that make this a pretty difficult task, and it’s the reason why the above three methods should not be used in isolation.  In combination, they help tell the whole story.

• It is difficult to know what users really read on a page.  In the first two methods you pretty much have to show people what to read — that doesn’t happen when they visit your site organically with no one looking over their shoulder.  This is why A/B testing is so important as it gives you a sense of how behavior will change based on content.
• It is difficult to isolate the effect of content changes from the other influencing factors on a page.  This is the really difficult part.  How do you know that conversion/engagement improved because of the content and not of some other factor on the page, like visual design changes?  That is why it is important to keep the rest of the page exactly the same, and also why usability and desirability testing is important to bring out the perceptual data from users.

And the biggest problem is of course that this is an idealistic approach.  Finding the resources/time/money to do this for every content change is obviously not feasible.  But for major changes to the site, this approach could be well worth the investment.

This is also by no means the only way to measure content effectiveness, but I think it’s a good approach that balances methodological rigor with the dangers of not overdoing it.  I’d be curious if anyone has any thoughts or ideas on how to improve on this approach…