Over the past few weeks, we’ve explored how there are three key principles that explain why site search just doesn’t perform like we expect it to and what we can consider to help mitigate this.  So far, we’ve learned that:

1. The critical information we need to make search great isn’t in the document – it’s in the users’ heads.
2. Asking users to explicitly provie us information that would improve search, while a seemingly good approach, is inherently flawed.

Today, we explore the third principle that shows that if we want to improve search, we need to focus on all of the things users are doing online.  What I mean here is that we need to look beyond search and at the entire site experience to truly understand what’s valuable, why it’s valuable, and in what context it’s found to be valuable.

Reason #3:  Search does not exist in a vacuum.

In order to improve search, we need to observe more than just search behavior.  Search and navigation have traditionally been seen as two separate paradigms: separate interfaces and separate systems driving them.  But in reality what’s happening?  A user is coming to your site and expressing an interest or intent through their actions.  They might have first expressed that intent through a Yahoo! or Google search that brought them to your site.  They might then express it in the pages they visit and engage with, the navigation they use, the links they click, and maybe the site searches they perform.  This expression of interest may span multiple searches and clicks.  And, finding documents that hold true value for that interest and intent may also take multiple steps.

Let’s think back to the “Insight/Incite” example once more.  Had we only looked at what search results users clicked on, the problem might never have been solved.  Why?  Because the valuable content was never in the results – it wasn’t there to be clicked on in the first place!  To learn what users really meant by “insight”, we had to watch their subsequent navigation, paying particular attention to the patterns of behavior that indicated engagement or that they had discovered content that was of value – even if it happened several steps after the initial search.  Observing search behavior alone is not enough!

What about users who don’t search at all?  What can we learn from them?  Users are actually giving us continual clues to their intent and interest with every link they click and every category they choose.  The documents that users engage with and the order in which they engage also tell us not only about relationships between documents, but intent.  If we take this valuable, implicit insight into account, then we really begin to see how this insight could be used to fix search.

What’s really remarkable is that once we take a step back and think of the entire online experience as a single unified expression of intent and value, we can do a lot more than fix search.  We can start to make recommendations and optimize the user experience with every interaction they make with your site; from the moment they arrive, every step they take through the site, as well as every search they perform. The true goal is to understand the user’s intent and then automatically surface documents that other like-minded peers have found valuable in that same context.  That’s the true wisdom of the crowd, and what Baynote’s Collective Intelligence Platform (CIP) is all about.

There are three key principles that explain why site search just doesn’t perform like we expect it to. Over the next few weeks, I’ll dive deeper into each issue surrounding traditional search and offer my insights and experiences to help you understand why your search sucks, and how you can improve it.

Reason #2:  Actions speak louder than words.

Okay, so we’ve figured out that the critical information is not in the document.  Where is it?  Well, it’s in the users’ heads of course.  Let’s look at an example.  If we look at a work by Shakespeare—or any great work of literature—the meaning cannot be identified simply by looking at the words within it.  It’s synthesized in the reader’s mind, and different readers may derive different meanings based on their own unique makeup and experiences.  The same applies to any document.  We must look beyond the words within the document to truly understand the value.  The key question to ask is this: when and why is this document valuable to users?  Only the users themselves know the answer.

Now that’s all well and good, but how do we extract that knowledge from the users?  We could ask them directly, but while that might seem like a good strategy, it’s actually not.  Asking users to explicitly rank, rate, or tag documents is doomed to failure.  The core problem is one of participation.  Think back to how many times you’ve provided feedback on the web.  Most of us never have; others may have on occasion, but almost certainly not on every page visit or search result.  This participation problem leads to a few key challenges:

A. Low coverage. A small subset of the population rates content, and when they do, the ratings only tend to  cover the most popular content.  Where does that leave the majority of our content?—the long tail.  Unranked and therefore undiscovered.  And with search, it’s not just about knowing that a document has value; it’s about whether it’s valuable for that specific search topic and that user.

Let’s take a deeper look at this.  Let’s say we have a bunch of ratings on a particular camera.  Sounds great, right?  Well, not really.  People like or dislike a camera for a variety of reasons.  Someone looking for a “lightweight camera” might think it stinks, while someone looking for a “cheap camera” might love it.  You can’t ignore the context of what a person is looking for, and getting explicit coverage across every topic of interest is even harder than just getting an overall rating.

And, when you factor in staleness of content — be it the article or the rating — then even the ratings you do have become less meaningful.  Add to that the mountains of content that are being created every day, and the problem really gets out of hand.  There’s just no way to keep up with it if we’re relying on people to go out of their way to explicitly rank, rate, and tag content.

B. Biased. In general, the people who do participate in explicitly rating something online represent a very small subset of the population.  That means that, even for those documents and topics that do have coverage, there’s no guarantee that the knowledge imparted by users even represents the majority opinion.  In fact, it almost always represents fringe opinions that are either extremely positive or negative because those are the people motivated to be heard.

To read more about the bias inherent in explicit methods of capturing community wisdom, check out my “7 Deadly Biases” whitepaper.

C. Inaccurate/Incomplete. Even when an individual decides to provide some form of explicit feedback, it often is not fully representative of even their own experience.  Let’s go back to the insight/incite example from my previous post.  Let’s say that a user decided to tag the Incite product page that was so useful.  What tag do you think they would use?  Probably “incite” or “incite phone,” right?  They would almost certainly not tag it with “insight” once they realized their mistake; but this is actually the tag that would be of most value to the community!  It might surprise you to know that social scientists are generally distrustful of people’s own accounts of their feelings and behavior!  Meta-cognition and emotional self-awareness are far more complex, and less intuitive, than people think.

So, if asking people to tell us what documents are valuable and why they’re of value doesn’t work, then what is the right way?  The key is to observe what people do, not what they say.  It’s both more accurate and more comprehensive.  The wisdom we are looking to tap is present in every single search–whether the searchers were successful or not.  By watching what people do, we can understand which documents are valuable and when.

But we need to be careful here too, because watching which search results users click on is not enough.  Clicks are a very weak indicator: just because people clicked on a result does not mean it’s valuable.  Perhaps the title was intriguing, confusing, or even misleading.  It is critical to follow the user all the way from query to success or ultimate failure—even if several steps later—and not get distracted by what they click on.  At Baynote, we track 24 different behavioral heuristics to ensure that we accurately capture where users are engaging given a particular context and intent.  And it’s not just about search; it’s about the entire online experience.  More on this in the next post.

Next week: Part 4 in the 4-part series, My Search Sucks! where we’ll explore how search does not exist in a vacuum.

My Search Sucks! Part 2

Part 2 in the 4-part series, “My Search Sucks,” discussing why search, well, sucks.

There are three key principles that explain why site search just doesn’t perform like we expect it to. Over the next few weeks, I’ll dive deeper into each issue surrounding traditional search and offer my insights and experiences to help you understand why your search sucks, and how you can improve it.

Reason #1:  The critical information is not in the document.

All full-text search technologies basically work the same way: they look for a match between the words in a user’s query and the words in the text of the documents searched.  That said, there are lots of fancy layers that can be added from simple stemming to complex natural language processing (NLP), but the fundamental assumption is that the engine can figure out which documents best meet a user’s needs by looking inside the document.

While this is a start, it’s just not enough. The critical information isn’t in the document; it’s in someone’s head.  But whose head is it in? Let’s look at some examples.

A favorite example comes to mind involving one of our customers, a large online appliance retailer.  Users were coming to their website looking for a “stove” over and over again, and the search results had “stove-top safe” kettles and pots, but no stoves.  Turns out the reason for this was that this retailer’s website was using the manufacturer terminology, “cooktops” and “ranges.”  The word “stove” was nowhere to be found.  The community was using a different vocabulary than the site.

Sounds like a simple fix, right?  All you need to do is to create a synonym to tell the search engine that a “stove” is the same thing as a “range.”  And sure, once you’ve found and addressed the discrepancy, customers searching for “stoves” will find the “ranges” they’re really looking for.  But what about all of those long-tail terms and content—and what about when things change?

Sam Mefford, an expert in the deployment of enterprise search technologies, commented on last week’s blog.  In his search practice, he sees this challenge surface on a regular basis and provided an example from one of his clients.  The company re-branded one of its products, and made the appropriate changes in its marketing materials and documentation.   Afterwards, field agents and customers could no longer find the products and information they needed, because they continued to search using the old name.  This problem took months to discover.

Another great example is from a customer that’s a well-known wireless provider.  They launched a new LG phone called the “Incite.”  Suddenly, one of the most popular search queries on their site became “insight.”  The search results included lots of business-type documents about how to achieve great “insight” into your business operations, but nothing that matched what users wanted – information on this exciting new phone. Sure, searching for “insight” while the product is called “incite” was technically the user’s mistake, but does that matter when you’re losing opportunities?

Let’s say the words do exist in the document.  It’s often not enough.  There may be 1000s of documents that contain the search terms, but which documents are the best?  A traditional search engine will assume that the one with the most occurrences of the keywords is the most valuable, but this is very often not the case.  Obviously, the technology is more sophisticated than this, but the fundamental basis is along these lines.  The most useful document may only have one instance of the keyword and therefore may be buried on page 10 of the results.  So, how do you get the most useful document to the top of the search results?

Manual tuning is the traditional “solution” to all of these site search issues, but as we discussed earlier, it’s nearly impossible to catch all discrepancies and adapt rapidly—not to mention the time and effort involved.  I’ve even mentioned the spirit of the solution: it’s fundamentally a recognition that the needed information is not in the document, it’s in someone’s head.  But whose head is it really in?

Many companies have experts that manually tune and tweak search.  But that’s a labor-intensive way to temporarily solve the problem and certainly doesn’t guarantee that the expert’s view on what’s right matches with users’.  Why take that chance?  Better to go straight to the source of the information: the user!

Next week: Part 3 in the 4-part series, My Search Sucks! where we’ll explore how actions speak louder than words.

1. The critical information is not in the document

While documents — whether webpages, pdfs, or Word docs — seem like the best place to discover a match to a user’s search term, they’re not. Processing documents is a good start, but the words within a document do not necessarily match the way a user understands the topic and phrases their question. And even if the search term is in there, it doesn’t mean that particular document is useful. The critical information is in the heads of users, not the documents. The key is to understand how, when, and why people use each document. At Baynote, we call this UseRank.

2. Actions speak louder than words

To get information from users you might think the best approach is to ask them. Seems simple and straightforward, right? Wrong. Turns out that there are a number of problems with explicit means of collecting information stemming from who participates, when, and why. As social science has taught us all along, if you really want to understand people, watch what they do, not what they say.

3. Search does not exist in a vacuum

Any time someone comes to your website, they are looking for something and they give you clues to what that is through both their search and navigation behaviors — and not just what they ask for and where they go, but what they do when they get there. Often they got to your site through an external link such as a search on the web — that’s your first clue. Although the goal might be to solve the site search problem, observing search behavior alone is not enough.

I’ll expand on each of these in more detail in upcoming posts.

The post, written by YouTube product manager Shiva Rajaraman, explains that the majority of YouTube users who rank videos give them a perfect 5-star ranking. He continues:

Seems like when it comes to ratings it’s pretty much all or nothing. Great videos prompt action; anything less prompts indifference. Thus, the ratings system is primarily being used as a seal of approval, not as an editorial indicator of what the community thinks about a video. Rating a video joins favoriting and sharing as a way to tell the world that this is something you love.

Rajarman goes on to solicit the community for feedback on how useful the current ranking system is and what can be done to improve upon it.

We’re really glad to see that YouTube is finally examining its rating system with an eye on delivering more value to its community and look forward to seeing how the system evolves from here. Ratings and user generated reviews, though often misleading, have become an expected part of the online experience and encourage deeper engagement. I don’t think anyone would take away points from YouTube on their ability to engage an incredibly large, diverse and influential community of users. However, YouTube’s review system- and others like it –  must also find ways to inform ratings based on valuable sentiment and implicit feedback gathered from the vast majority of their site visitors. Not the loud minority.

With a truly integrated approach to recommendations that blends both implicit and explicit feedback, companies can expect to improve engagement and overall user experience by directing site visitors to the best content based on their intent.  I talk a lot about this concept in my paper, entitled “7 Deadly Biases”.

In the end, explicit versus implicit user feedback shouldn’t be viewed as an either/or scenario. Please let us know your thoughts on the matter and share examples of sites that are doing it right.