Part 3 in the 4-part series, “My Search Sucks,” discussing why search, well, sucks.

There are three key principles that explain why site search just doesn’t perform like we expect it to. Over the next few weeks, I’ll dive deeper into each issue surrounding traditional search and offer my insights and experiences to help you understand why your search sucks, and how you can improve it.

Reason #2:  Actions speak louder than words.

Okay, so we’ve figured out that the critical information is not in the document.  Where is it?  Well, it’s in the users’ heads of course.  Let’s look at an example.  If we look at a work by Shakespeare—or any great work of literature—the meaning cannot be identified simply by looking at the words within it.  It’s synthesized in the reader’s mind, and different readers may derive different meanings based on their own unique makeup and experiences.  The same applies to any document.  We must look beyond the words within the document to truly understand the value.  The key question to ask is this: when and why is this document valuable to users?  Only the users themselves know the answer.

Now that’s all well and good, but how do we extract that knowledge from the users?  We could ask them directly, but while that might seem like a good strategy, it’s actually not.  Asking users to explicitly rank, rate, or tag documents is doomed to failure.  The core problem is one of participation.  Think back to how many times you’ve provided feedback on the web.  Most of us never have; others may have on occasion, but almost certainly not on every page visit or search result.  This participation problem leads to a few key challenges:

A. Low coverage. A small subset of the population rates content, and when they do, the ratings only tend to  cover the most popular content.  Where does that leave the majority of our content?—the long tail.  Unranked and therefore undiscovered.  And with search, it’s not just about knowing that a document has value; it’s about whether it’s valuable for that specific search topic and that user.

Let’s take a deeper look at this.  Let’s say we have a bunch of ratings on a particular camera.  Sounds great, right?  Well, not really.  People like or dislike a camera for a variety of reasons.  Someone looking for a “lightweight camera” might think it stinks, while someone looking for a “cheap camera” might love it.  You can’t ignore the context of what a person is looking for, and getting explicit coverage across every topic of interest is even harder than just getting an overall rating.

And, when you factor in staleness of content — be it the article or the rating — then even the ratings you do have become less meaningful.  Add to that the mountains of content that are being created every day, and the problem really gets out of hand.  There’s just no way to keep up with it if we’re relying on people to go out of their way to explicitly rank, rate, and tag content.

B. Biased. In general, the people who do participate in explicitly rating something online represent a very small subset of the population.  That means that, even for those documents and topics that do have coverage, there’s no guarantee that the knowledge imparted by users even represents the majority opinion.  In fact, it almost always represents fringe opinions that are either extremely positive or negative because those are the people motivated to be heard.

To read more about the bias inherent in explicit methods of capturing community wisdom, check out my “7 Deadly Biases” whitepaper.

C. Inaccurate/Incomplete. Even when an individual decides to provide some form of explicit feedback, it often is not fully representative of even their own experience.  Let’s go back to the insight/incite example from my previous post.  Let’s say that a user decided to tag the Incite product page that was so useful.  What tag do you think they would use?  Probably “incite” or “incite phone,” right?  They would almost certainly not tag it with “insight” once they realized their mistake; but this is actually the tag that would be of most value to the community!  It might surprise you to know that social scientists are generally distrustful of people’s own accounts of their feelings and behavior!  Meta-cognition and emotional self-awareness are far more complex, and less intuitive, than people think.

So, if asking people to tell us what documents are valuable and why they’re of value doesn’t work, then what is the right way?  The key is to observe what people do, not what they say.  It’s both more accurate and more comprehensive.  The wisdom we are looking to tap is present in every single search–whether the searchers were successful or not.  By watching what people do, we can understand which documents are valuable and when.

But we need to be careful here too, because watching which search results users click on is not enough.  Clicks are a very weak indicator: just because people clicked on a result does not mean it’s valuable.  Perhaps the title was intriguing, confusing, or even misleading.  It is critical to follow the user all the way from query to success or ultimate failure—even if several steps later—and not get distracted by what they click on.  At Baynote, we track 24 different behavioral heuristics to ensure that we accurately capture where users are engaging given a particular context and intent.  And it’s not just about search; it’s about the entire online experience.  More on this in the next post.

Next week: Part 4 in the 4-part series, My Search Sucks! where we’ll explore how search does not exist in a vacuum.

My Search Sucks! Part 2

Part 2 in the 4-part series, “My Search Sucks,” discussing why search, well, sucks.

There are three key principles that explain why site search just doesn’t perform like we expect it to. Over the next few weeks, I’ll dive deeper into each issue surrounding traditional search and offer my insights and experiences to help you understand why your search sucks, and how you can improve it.

Reason #1:  The critical information is not in the document.

All full-text search technologies basically work the same way: they look for a match between the words in a user’s query and the words in the text of the documents searched.  That said, there are lots of fancy layers that can be added from simple stemming to complex natural language processing (NLP), but the fundamental assumption is that the engine can figure out which documents best meet a user’s needs by looking inside the document.

While this is a start, it’s just not enough. The critical information isn’t in the document; it’s in someone’s head.  But whose head is it in? Let’s look at some examples.

A favorite example comes to mind involving one of our customers, a large online appliance retailer.  Users were coming to their website looking for a “stove” over and over again, and the search results had “stove-top safe” kettles and pots, but no stoves.  Turns out the reason for this was that this retailer’s website was using the manufacturer terminology, “cooktops” and “ranges.”  The word “stove” was nowhere to be found.  The community was using a different vocabulary than the site.

Sounds like a simple fix, right?  All you need to do is to create a synonym to tell the search engine that a “stove” is the same thing as a “range.”  And sure, once you’ve found and addressed the discrepancy, customers searching for “stoves” will find the “ranges” they’re really looking for.  But what about all of those long-tail terms and content—and what about when things change?

Sam Mefford, an expert in the deployment of enterprise search technologies, commented on last week’s blog.  In his search practice, he sees this challenge surface on a regular basis and provided an example from one of his clients.  The company re-branded one of its products, and made the appropriate changes in its marketing materials and documentation.   Afterwards, field agents and customers could no longer find the products and information they needed, because they continued to search using the old name.  This problem took months to discover.

Another great example is from a customer that’s a well-known wireless provider.  They launched a new LG phone called the “Incite.”  Suddenly, one of the most popular search queries on their site became “insight.”  The search results included lots of business-type documents about how to achieve great “insight” into your business operations, but nothing that matched what users wanted – information on this exciting new phone. Sure, searching for “insight” while the product is called “incite” was technically the user’s mistake, but does that matter when you’re losing opportunities?

Let’s say the words do exist in the document.  It’s often not enough.  There may be 1000s of documents that contain the search terms, but which documents are the best?  A traditional search engine will assume that the one with the most occurrences of the keywords is the most valuable, but this is very often not the case.  Obviously, the technology is more sophisticated than this, but the fundamental basis is along these lines.  The most useful document may only have one instance of the keyword and therefore may be buried on page 10 of the results.  So, how do you get the most useful document to the top of the search results?

Manual tuning is the traditional “solution” to all of these site search issues, but as we discussed earlier, it’s nearly impossible to catch all discrepancies and adapt rapidly—not to mention the time and effort involved.  I’ve even mentioned the spirit of the solution: it’s fundamentally a recognition that the needed information is not in the document, it’s in someone’s head.  But whose head is it really in?

Many companies have experts that manually tune and tweak search.  But that’s a labor-intensive way to temporarily solve the problem and certainly doesn’t guarantee that the expert’s view on what’s right matches with users’.  Why take that chance?  Better to go straight to the source of the information: the user!

Next week: Part 3 in the 4-part series, My Search Sucks! where we’ll explore how actions speak louder than words.

The central focus of Gartner’s Symposium/ITxpo this week in Orlando is all about implementing what they’ve recently dubbed as a “pattern-based strategy”.

According to Gartner, a pattern-based strategy “provides a framework to proactively seek, model and adapt to leading indicators, often-termed ‘weak’ signals that form patterns in the marketplace.”  For the past several years Baynote has been committed to helping companies identify these patterns with technology that lets them tap into the collective intelligence of customers visiting their websites. This is something that transactional based systems such as business intelligence (BI) and complex event processing (CEP) simply haven’t been able to deliver. Here’s why:

1) For years BI, CEP (more recently) and other related technologies have helped organizations become much more efficient by automating their interactions with customers. However, in the process of creating huge economies of scale, they forced companies to lose the “mom and pop” touch that consumers expect when they walk into a local hardware store or restaurant. In failing to create digital mom and pop experiences, online retailers and publishers have placed unnecessary emphasis on promoting popular products and content, thereby losing out on profits to be gained from merchandising their long tail products.

2) In addition, these so-called “predictive” applications have historically prioritized the wrong set of indicators, often identifying consumer trends weeks, if not months, after the fact. For example, e-commerce transactions lag other more relevant indicators, such as online comparison shopping, by months. Only by tapping into the power of the collective is it possible to see early signals, spot trends and develop strategies around them before your competitors catch on. This holds particularly true for long tail products. Our customer US-Appliance tapped into the implicit behaviors of its website visitors to merchandise colored washers/dryers months before Home Depot and Best Buy began promoting similar products in their stores.

In Gartner’s recent report, entitled “Introducing Pattern-Based Strategy”, they view “the collective” as being critical to developing a pattern-based strategy. We couldn’t agree more with their position:

The collective comprises individuals, groups, communities, mobs, markets and firms that shape the direction of society and business. The collective is not new but technology has made the collective more powerful — and enabled change to happen more rapidly. The explosion of social software has enabled groups and individuals to rapidly form and rally to a cause — often resulting in significant societal changes.

The result for business is a cacophony of rapidly evolving demands, expectations, inputs and transactions, as well as an opportunity to not only react, but to seek signals of change from the collective. Market trends, some subtle, others strong, are masked by noise, and many enterprises are failing to proactively detect the patterns they rely on to direct future strategy and support investment decisions. In addition to failing to detect these patterns, enterprises are not utilizing new resources to proactively seek signals of change nor do they understand their power to influence individuals and communities.

Val Sribar, group vice president of Research at Gartner, sites Amazon’s and Netflix’s use of recommendation engines as good examples of organizations leveraging collective intelligence to support their pattern-based strategies. Sribar agrees with Baynote that recommendation engines identify new patterns in behavior as customers browse and purchase. While Amazon and Netflix are highly popularized cases, we’ve helped hundreds of other well known brands tap into their collective customer networks to significantly increase revenue through cross-selling and upselling, and higher customer loyalty.

We’re excited to see Gartner take a leadership position on this important issue and look forward to working with them and our customers to bring best practices related to collective intelligence to the forefront of modern business strategy.

“My Search Sucks!” — we hear this from prospects more than any other complaint. Coming from consumer search experiences on the web with the likes of Google, Yahoo, and new entrant Bing, these frustrated employees wonder why they can’t get better search results on their company’s website and intranet. Fair question. Turns out there are a few key principles that explain why site search often sucks and how to fix it:

1. The critical information is not in the document

While documents — whether webpages, pdfs, or Word docs — seem like the best place to discover a match to a user’s search term, they’re not. Processing documents is a good start, but the words within a document do not necessarily match the way a user understands the topic and phrases their question. And even if the search term is in there, it doesn’t mean that particular document is useful. The critical information is in the heads of users, not the documents. The key is to understand how, when, and why people use each document. At Baynote, we call this UseRank.

2. Actions speak louder than words

To get information from users you might think the best approach is to ask them. Seems simple and straightforward, right? Wrong. Turns out that there are a number of problems with explicit means of collecting information stemming from who participates, when, and why. As social science has taught us all along, if you really want to understand people, watch what they do, not what they say.

3. Search does not exist in a vacuum

Any time someone comes to your website, they are looking for something and they give you clues to what that is through both their search and navigation behaviors — and not just what they ask for and where they go, but what they do when they get there. Often they got to your site through an external link such as a search on the web — that’s your first clue. Although the goal might be to solve the site search problem, observing search behavior alone is not enough.

I’ll expand on each of these in more detail in upcoming posts.

You’ve probably caught some of the hype about NASA’s latest Lunar exploration efforts. This morning they deliberately crashed a spent upper stage rocket booster into the moon in order to analyze the debris plume for traces of water. Exciting stuff in its own right, but here at Baynote we always get a little bit extra excited when NASA does something interesting.

You see, the NASA.gov website uses Baynote to understand its visitors’ true intent and produce social search results as well as content and video recommendations that are most appropriate to each individual visitor. So every time NASA launches a manned mission or deploys a rover or slams something into the lunar surface at twice the speed of a bullet, a little bit of Baynote is out there, too.

This morning’s culmination of the LCROSS mission marked the second busiest day NASA’s website has ever weathered, with millions of concurrent users around the world watching live video and reading about the mission, and Baynote’s recommendations never missed a beat. So congratulations to NASA on another job well done, and thanks for taking us along for the ride.

According to a new consumer privacy study by the Berkeley Center for Law & Technology at UC Berkeley, and the Annenberg School for Communication at the University of Pennsylvania, two thirds of Americans object to online tracking by advertisers. The study is apparently the first national telephone survey that explores Americans’ opinions about the controversial practice of behavioral targeting.  Here’s a statement from the press release about the report, which was issued on Wednesday:

The report, Americans Reject Tailored Advertising shows that 66 percent of adults said no to tailored ads. Not only that, when informed of specific behavioral targeting techniques that marketers employ to create the ads, even higher percentages — between 73 percent and 86 percent — oppose tailored advertising. Those techniques include tracking behavior on websites and in retail stores.

For a more detailed analysis of the findings, you should check out Stefanie Clifford’s coverage of the report in the New York Times.

We’ve been talking about the pitfalls of behavioral targeting for years, so it’s nice to finally see some national research that tells marketers what consumers actually think about this shady technique. In this age of identity theft and mounting concerns over privacy in general, a practice that proactively profiles a user — perhaps over the scope of many Web sites and over a period of several months — will sound alarms even among the least conservative of us.

Beyond privacy concerns, there are bigger issues with behavioral targeting related to accuracy and quality, that many marketers still don’t understand. Traditional behavioral targeting struggles precisely because it tries to discern what I want now based on my past behaviors. Consider the impact of focusing on historical interests instead of current intent: If I bought a gift for my niece on Amazon.com last week, I certainly don’t want to be bombarded by ads for similar products that probably aren’t relevant during my next visit.

Another way to think of this problem is to consider the idea of roles or what personalization systems might call “profiles”. Humans have far too many roles in life for a profile to possibly predict what a user wants on any given day. A woman shopping for baby clothes, a tie for her husband, and a gift for her sister may appear schizophrenic because she is acting in three different roles — mother, wife and sister. What do you show her next? Tossing strollers ads at her isn’t going to be effective now that she’s shopping for a new cocktail dress for herself.

This is the pitfall of profiles. In a given month, an individual will have thousands of roles. Knowing my past is not necessarily a better way to predict my future. In fact, this phenomenon has been known by psychologists and other scientists for years — humans are animals of context and situations, much less than of historical profiles or roles.

Enter Intent-driven Targeting

An alternative that solves the issues with both privacy and effectiveness is one centered on understanding the users’ intentions, instead of their clickpaths or profiles, and pairing that knowledge with specific content, product and advertising recommendations. This approach relies exclusively on the collective wisdom of like-minded peers who have demonstrated interests or engagement with similar content and contexts.

The concept of profiles is completely removed in this case. Instead, through understanding expressed or implied intent, content appropriate to the user’s current mindset can be delivered.

Most importantly, it kills two birds with one stone: Users get useful, accurate recommendations and ads, while still avoiding the whole privacy mess.

I admit it; I’m just not interested enough in my toaster to leave a review for others. My intuition tells me that I’m probably not alone; there are few things about my toaster that compel me to share my experience with others.

Statistically, a very small percentage of any web community will take the time to leave explicit feedback for anything on a website. Most web visitors represent a “silent majority” who regularly engage with a website but do not necessarily feel passionate enough about a particular product to share their opinion with others. As a result, the power of communicating sentiment regarding products is left to the vocal minority of a web community that will take the time to actually leave ratings and reviews.

An alternative method for delivering a more “balanced” and “representative” form of personalization would be to utilize implicit heuristics which factor in 100% of a visitor’s browsing experience. Implicit systems do not require visitors to explicitly rate or review anything; instead, they deliver personalization by simply observing implicit usage heuristics such as behavioral patterns or interactions exhibited during a site experience which communicates genuine engagement and interaction with particular products.

The reality is that most products do not inspire legions of visitors to contribute ratings or reviews in the hopes of finding love, fortune, and glory. The reality is that if someone is selling toasters, appliances, and most other “everyday” products, a more comprehensive and useful approach to personalization may be involve letting actions speak louder than words.

Richard MacManus over at ReadWriteWeb recently turned me on to an interesting YouTube blog post about the effectiveness of the popular video aggregator’s 5-star rating system.

The post, written by YouTube product manager Shiva Rajaraman, explains that the majority of YouTube users who rank videos give them a perfect 5-star ranking. He continues:

Seems like when it comes to ratings it’s pretty much all or nothing. Great videos prompt action; anything less prompts indifference. Thus, the ratings system is primarily being used as a seal of approval, not as an editorial indicator of what the community thinks about a video. Rating a video joins favoriting and sharing as a way to tell the world that this is something you love.

Rajarman goes on to solicit the community for feedback on how useful the current ranking system is and what can be done to improve upon it.

We’re really glad to see that YouTube is finally examining its rating system with an eye on delivering more value to its community and look forward to seeing how the system evolves from here. Ratings and user generated reviews, though often misleading, have become an expected part of the online experience and encourage deeper engagement. I don’t think anyone would take away points from YouTube on their ability to engage an incredibly large, diverse and influential community of users. However, YouTube’s review system- and others like it –  must also find ways to inform ratings based on valuable sentiment and implicit feedback gathered from the vast majority of their site visitors. Not the loud minority.

With a truly integrated approach to recommendations that blends both implicit and explicit feedback, companies can expect to improve engagement and overall user experience by directing site visitors to the best content based on their intent.  I talk a lot about this concept in my paper, entitled “7 Deadly Biases”.

In the end, explicit versus implicit user feedback shouldn’t be viewed as an either/or scenario. Please let us know your thoughts on the matter and share examples of sites that are doing it right.

Recommendations offer a key driver for unlocking the value of email marketing campaigns. By adding email recommendations you provide your customers a compelling reason to continue their shopping experience, and drive significant gains in response rates and sales conversion.

Watch the full webinar now.

This webcast features Forrester Analyst Julie Katz, author of “Your Email Marketing Road Map For 2009″ and Jay DeFoore, Director of Consumer Marketing at The Knot Inc. Julie and Jay will discuss how marketers can make the most of available tools and services to open doors for radical innovation in the channel.

Ms. Katz will reveal some of Forrester’s latest findings in her “How To Improve Email In 2009″ series, including 10 ways email marketers can get more from their programs today.

In this joint Baynote/Responsys webinar you will also find out how to:

• Target your emails based on observed website visitor behavior and interests
• Deliver emails at optimal times when customers are most receptive
• Measure the impact of your email marketing initiatives from email open rates to conversion and sales
• Test and Optimize to continually improve conversion results and ROI

Watch the full webinar now.

It has been a few days of surreal experience. Baynote has lost a truly trusted adviser, expert and friend. The loss is tragic, deep and personal because it is just not Rajeev’s time to go yet. He is too young, too smart, and with too good of a heart not to be here with us in Silicon Valley and with Baynote.

Rajeev was instrumental to the success of Baynote from the very beginning, even before we had a name for our company. He advised us why Google is a partner instead of a potential competitor. He offered many unique insights to the technology and business in the last five years. Two weeks ago, we had the fortune and luck to sync up with him one last time. He was so live, full of energy and ideas. Our one hour Palo Alto coffee talk went on and on. He was so excited about his planned trip to Wimbledon with his family this summer… And as usual, he was there to give unconditional help and made 7 connections after our chat…

Our thoughts and the thoughts of entire Baynote team, advisers and board go to Asha and the kids. Rajeev will be greatly missed physically but he will live on in our heart, spirit and via our technology!

Jack, Rob and Scott