Measuring your content with user data

I wanted to build on Keri’s recent insightful post on measuring content strategy, and talk about possible ways to measure the effectiveness of content changes on e-commerce sites.  More specifically, how do you select the best content if you have a variety of different alternatives, each with its own group of fans who want to get it on the site right away?  Since the voice of a web site can be such an abstract, arbitrary decision, how can we apply methodologically robust research methods to help make these decisions?

First, I would define “effectiveness” in this context as the optimization of the following three concepts:

• Do users understand what you are trying to tell them and what action they should take to be successful in their task?
• Are you invoking the desired emotions with your content?
• Does the proposed content result in higher conversion rates than other alternatives?

It’s so important to combine the user perception data (the first two bullets) with business metrics (the last bullet).  From my experience the only way for user experience professionals to affect change is if we can show the positive impact these changes have on engagement/revenue metrics.

It seems to me that you will be well served by using the following three methodologies to measure the relative effectiveness of different versions of the same content.  This is also a really nice way to progressively reduce the number of alternatives down to the best solution:

• Usability testing. Start with several different version of the content (~10), along with the current version (if it exists).  Ask users in a lab setting what they understand the content to mean, and any other thoughts they have on the way it sounds.  This should help narrow down the alternatives to 4-6 possibilities.
• Desirability testing.  Use the Desirability method, but adjust it for use in large sample online surveys by turning it into a between-subjects experimental design.  In the survey, users are asked to rate the content on different brand and design attributes.  This way you can determine what emotional response the content extracts out of users.  You’d also be able to ask users which version of the content they’d prefer, and why.  This method has the added benefit of large numbers to give you confidence in the statistical significance of the results.
• A/B testing.  Once you’ve narrowed the alternatives down to two or three, live A/B testing can help you determine which of the alternatives perform better from a revenue or engagement perspective, by looking at differences that can be attributed purely to content changes.  This obviously works easiest when the content is directly related to a revenue-generating task, like the call to action on a checkout page, for example.  But it’s not just about revenue — there are great ways to measure metrics of engagement with the page, which is just as powerful.

Now, I can see two issues that make this a pretty difficult task, and it’s the reason why the above three methods should not be used in isolation.  In combination, they help tell the whole story.

• It is difficult to know what users really read on a page.  In the first two methods you pretty much have to show people what to read — that doesn’t happen when they visit your site organically with no one looking over their shoulder.  This is why A/B testing is so important as it gives you a sense of how behavior will change based on content.
• It is difficult to isolate the effect of content changes from the other influencing factors on a page.  This is the really difficult part.  How do you know that conversion/engagement improved because of the content and not of some other factor on the page, like visual design changes?  That is why it is important to keep the rest of the page exactly the same, and also why usability and desirability testing is important to bring out the perceptual data from users.

And the biggest problem is of course that this is an idealistic approach.  Finding the resources/time/money to do this for every content change is obviously not feasible.  But for major changes to the site, this approach could be well worth the investment.

This is also by no means the only way to measure content effectiveness, but I think it’s a good approach that balances methodological rigor with the dangers of not overdoing it.  I’d be curious if anyone has any thoughts or ideas on how to improve on this approach…

Champagne and Wii, not made for each other

Keri and I had an interesting discussion last week.  Well, I think it was interesting. I believe she thought it was pretty stupid.  We’ll let you be the judge.

It started with a seemingly innocent story she told about having champagne while playing Wii.  I would like to lay out my argument here for why I think champagne should not be consumed while playing Wii.  It’s like this:

• Champagne is a pretentious drink.  It is all bubbly and alcoholly, like a beauty queen at a party.  Champagne is aloof, as if it’s better than all the other drinks.  It sometimes hangs out with less fancy drinks like orange juice, but that’s just like celebrities doing charity work.  It’s all about appearances with champagne.
• Wii is a down-to-earth, everyday, life-is-good type of activity.  Wii is the guy who shows up at the party and brings the music everyone has been waiting for.
• These 2 activities don’t mix.  Champagne is for the artifical inflation of events like baby showers and brunch at a fancy hotel.  Wii is for celebrating the simplicity of life.  Consuming the two together sends a confused message that our brains shouldn’t have to be subjected to.

Why am I posting this in a content blog?  Another conversation Keri and I had recently was about the role of tone in representing meaning.  I think part of my (admittedly tantrum-like) view on the champagne vs. Wii issue, is the words and what they represent.

Champagne has an elitist sound to it (but ironically comes from a French word that means “open country”).  Wii, on the other hand, is a totally made up word meant to represent fun (and also work in multiple languages – see this article on the history behind this).  Words have personas well beyond their meaning.  I guess I just want those personas to mix with their own kind.

Making up twords

I guess we should have seen it coming when e-commerce first became a word.  E-business, E-marketing, E-mail, E-vites (and the inevitible bastardizations – “how do you un-evite someone??”).

Next came the i-words.  iPod, iPhone, iHome, iRanOutOfBrandNamesSoIPrefixEverythingWithI.

But I guess those are still all reasonably ok.  It’s this Twitter thing that finally threw us off a cliff and into an endless abyss of grotesque word-concoctions (in my humble opinion, of course).  I mean, really.  Do we need a “twictionary”?  Well, whether we need one or not, apparently a bunch of people cared enough to make one:
http://twictionary.pbwiki.com

So what’s your least favorite tword?  I’m gonna go with Puntwitocracy.  What a blotch on the English language.

And while we’re on the subject of Twitter.  I’m assuming everyone has seen this video by now.  But it’s still good to laugh at ourselves – especially since @deliciouswords is pretty active over there in the twitverse…

Highways, pangrams, palindromes

I know it’s a bit lame to write a post that just refers to someone else’s post, but this one is too good to let slip through the cracks.  After you go to Dictionary.com and look up the word pangram, you have to read this post about Interstate 287 in New Jersey:

A 0.8-mile stretch of northbound Interstate 287 in New Jersey contains these signs:

WASHINGTON’S HEADQUARTERS
NO TRUCKS IN LEFT LANE
LAFAYETTE AVE.
EXIT 20 MPH
BRIDGE FREEZES BEFORE ROAD SURFACE
INTERSTATE NEW JERSEY 287

To date this is the shortest reported stretch of U.S. highway whose permanent, official signs contain all 26 letters of the alphabet.

Of course a whole slew of questions come to mind:

1. Was this deliberate or an accident?
2. Did someone try to break the coveted “shortest pangram highway” record?  What’s the prize?
3. If this wasn’t deliberate, who counted the distance, the signs, and the letters?  Whoever it was, bless their hearts.

Speaking of “pa-” words…  What’s your favorite palindrome?  Mine has to be “A man, a plan, a canal – Panama.“  Can you beat that?

Blagojevich, The Solipsist

Sometimes I wish I was a journalist.  The dream usually hits me when I read essays that manage to not just report the news, but also put it in the proper sociatal context.  James Poniewozik did this recently in a brillant article on Ex-Governor Rod Blagojavich’s recent media blitz, and what it tells us about society (and reality TV!).  You can read it here.

The reason I’m writing about it is that Poniewozik (what a fun word to say…) used a word to describe Blagojevich that I didn’t know: solipsist.  In case you don’t know it either, solipsism is defined as:

1.     The theory that only the self exists, or can be proved to exist.
2.     Extreme preoccupation with and indulgence of one’s feelings, desires, etc.; egoistic self-absorption.

The philosophy, the word, and its usage to describe Blagojevich really intruiged me, so I dug a little deeper.  Here’s what I found on the logic (?) behind this philosophy:

I cannot be sure anyone else feels any emotions that I feel.  Since our only source of knowledge of the world is our personal senses (eyes, ears, etc.), it could be argued that one only knows that he or she exists. Is it possible that our senses betray us, and that all we see and hear are but tricks played on us by our mind?

Very Matrix-like.  The etymology of the word is also interesting:

Latin solus “alone” +ipse “self” + -ism. “Solus” is the origin of “solo” but also can be found in “solitary,” “soliloquy,” and “desolate.” In the Germanic languages it became solein “alone,” devolving into English “sullen.”

I like the word sullen, and didn’t know it came from the same root.  Very interesting.  Anyway, back to why I’m writing about it here.  Two reasons, really.

One, I feel it is important for Keri and I to teach you new words that you can throw into everyday conversation to feel smart.  This particular word has an added bonus in that you’d probably be able to insult someone without them knowing it.  It’s always handy to have a word like that lying around.

Second, I’m surprised that Poniewozik (hmm, it gets funnier each time you say it) used this word in a major magazine article.  I’m sure most people just glossed over it and didn’t think about it again.  But I applaud Mr. Poniewozik (he he) for not selling out and using a lesser word when solipsist is clearly the absolute best fit for what he’s trying to say.  So let’s take that as a lesson.  When faced with a choice between using a perfect word or dumbing it down – use the perfect word.  That’s how language stays alive and vibrant.

Defiantly reaching out

Yesterday I got an email from a vendor that said, among other things, the following:

we will defiantly reach out if we have any questions

It’s obviously an honest mistake, and all I should have done was chuckle and move on.  But I didn’t.  First I started thinking about how one would defiantly reach out to someone else with questions.  Perhaps, “I have a question, and I don’t care what your opinion is about it, but you WILL answer me!”?  I’m not sure.

And then I sent it to Keri, because I just wasn’t ready to be done with it.  She took it one step further, so I feel like it’s appropriate to copy her exact email here:

Two things:

“Defiantly.” Hilarious.

“Reaching out” as a term for “contact” or “call” or “email.” That really annoys me. “Why don’t you reach out to Chris and get more information?” No. No, I will NOT “reach out” to Chris. I don’t need Chris to save me and Chris doesn’t need counseling from me. I just need to CONTACT him.

Just say CONTACT.

AHHHHHHHHHHHHHH!

…Okay. I’m better now.

So clearly Keri has some issues.  But aside from that, it sparked a discussion on the use of fancy business jargon when good old-fashioned regular words would actually work just fine:

Reach out should be Contact
Leverage should be Use
Solve (as a noun) should be Solution
Touch base should Talk to
Absolutely should be Yes

And the list goes on and on.  So I ask of our tens of blog readers – what other phrases are we missing?  What business jargon drives you nuts?

By the way, to top it all off, this morning I received a widely distributed email where the author thanked a bunch of people for “breasting this tsunami project despite the aggressive timeline and other adverse conditions.”

I really have no idea how you would “breast a project,” so if anyone has any ideas about what that might mean, let us know…

I’m with it. I’m hip.

I was thinking about the word “hip” (as you do), and about the irony of such a decidedly unfashionable word for what the dictionary defines as “familiar with or informed about the latest ideas, styles, developments, etc.”

I’m not quite sure when exactly “hip” became a word your parents use to refer to something they don’t understand or recognize, but I’d like to think that the tipping point was Dr. Evil’s outcry in Austin Powers (”I’m with it. I’m hip. Tucka, Tucka, Tucka…Huhhhhh. Well don’t look at me like I’m freakin’ Frankenstein, give your father a hug.”).

The reality is probably a little more mundane. Words go in and out of fashion, and it’s hard to predict what will happen.  I am personally amazed at the longevity of the word “cool.”  How is it still ok to say that?  Paris Hilton brought “That’s hot” back, so there’s one easy explanation, at least.

Other words have shorter seasons.  The New York Times recently published a fantastic article about the buzzwords of 2008 – I bet “Caribou Barbie” is already forgotten, but words like “fail” and “staycation” are probably here to stay, for better or worse.

Back to “hip” though, and why I’m thinking about it (because I feel like I owe you an explanation). I always cringe when I see the programming summary that Comcast uses for The Daily Show: “Jon Stewart’s hip, irreverent skewering of the news and pop culture.” It bothers me because while irreverent is a perfect word to describe the show (”satirical, critical of what is generally accepted or respected”), hip is such a death sentence for coolness.

What other words for “cool” fell out of fashion? Am I missing anything?

What’s an Elegiac?

This morning an interesting headline caught my eye on the New York Times web site:

In Final News Conference, Bush Strikes Elegiac Tone

(No, REALLY, this is a blog about language, not politics!)

I have to confess that I had no idea what “Elegiac” means, so I looked it up. The first definition I found was a little less than satisfactory:

Noting a distich or couplet the first line of which is a dactylic hexameter and the second a pentameter, or a verse differing from the hexameter by suppression of the arsis or metrically unaccented part of the third and the sixth foot.

Since this definition introduced more words that I don’t understand, it seemed like a step backwards in my quest for knowledge, so I ignored it. Instead of giving up though, I persevered and finally figured out what “elegiac” means:

1. used in, suitable for, or resembling an elegy.
2. expressing sorrow or lamentation: elegiac strains.

An elegy, as I’m sure you know, is “a mournful, melancholy, or plaintive poem, especially a funeral song or a lament for the dead.”

So coming back to the Times article – this headline strikes me as odd for two reasons:

1. No one died
2. Press conferences aren’t exactly poetic (See how I generalized? I told you this isn’t a political blog.)

Unless, of course, the author was aiming a little higher than the literal meaning of the word, and believes that President Bush was lamenting (in a sorrowful tone) the demise of his presidency in its final weeks. Well, yes actually, that sounds like a pretty plausible explanation. Which would make this blog post a little bit of a waste. But am I going to delete it? Not so much. (Don’t you hate question-talking? I do too.)

But then again, it may just be that the author’s Word thesaurus threw out “elegiac” as a synonymn for “sad”, and he was under a tight deadline which didn’t leave him time to look it up. You never know with these things.

Peeance Freeance

No, really – this is not a political blog.  It’s a language blog.  But you can’t have politics without language, right?  So this recent CNN report on “Bushisms” deserves a mention (if the video doesn’t work in Firefox, try it in Internet Explorer):

So I guess now is as good a time as any to mention Going Nucular, one of our favorite books on the use (and misuse) of the English language:

How to pronounciate words

Do you remember Joe the Plumber?  If you don’t, I recommend you go do a quick Wikipedia search.  Go ahead, I’ll wait.

Ok, now that we’re on the same page.  So anyway, Joe is back.  In a staggeringly tone deaf move, “news” outlet Pajamas TV is sending Joe the Plumber to Israel to be war correspondent for their online division (and that’s all I’ll say about that, since this isn’t a political blog).  In an interview with Fox News yesterday, Joe said this to answer a question on how he is preparing himself for the trip:

Ah you know I’ve just been studying new, everything that comes out on the news lately, um, trying to get uh, how to pronounciate some of the names, things like that.

This sentence is a word geek’s dream.  There’s the “you know” indicating he doesn’t really know.  The “everything that comes out on the news lately” indicating “nothing that came out of the news”, and which is particularly ironic since he said earlier in the interview his reason for going there is this:

We don’t really get the full story. And so I’m gonna’ go there, interview some Israeli soldiers, civilian population and get their take on what’s going on.

But I digress.  There’s also the obligatory uh’s and um’s and “things like that” as he tries to come up with something remotely coherent.  And then of course, there is the pinnacle of content irony, the zinger: learning to pronounciate some of the names.

Classic.

For Jon Stewart’s take on all of this, see below (starting at 3:25).