P&G’s eStore was prompted by a mediocre year in online sales for the consumer goods manufacturer. According to a recent article by Andy McMains of AdWeek, less than one percent, or $500 million, of P&G’s $79 billion in global revenue last year stemmed from online sales via sites such as walmart.com and amazon.com. Through the eStore, P&G hopes to learn from consumers how to better market their products and improve online sales across all e-commerce channels. One of the main goals of this initiative is for P&G to learn about consumer buying patterns and share this customer intelligence with its retailer partners to boost sales.

eCommerce is unique in that Websites allow retailers to quickly identify consumer shopping patterns, and automatically adapt their merchandising programs based on this insight. This is simply not possible in brick-and-mortar stores. With Baynote’s technology, for example, retailers such as Urbanoutfitters.com and Bluefly.com are able to more quickly recognize even the most unexpected trends, such as consumers purchasing two seemingly unrelated products, and then personalize product recommendations in real-time. This improves the next shopper’s experience, builds loyalty and boosts sales.

Since most products are currently sold via retailers rather than directly by manufacturers, manufacturers are often left in the dark on the valuable data that their retail partners can obtain about consumer behavior. While brands have been able to utilize the Internet well to engage with consumers via email marketing, social networks, online product reviews, and a slew of other activities that require their explicit feedback, they have not had a direct link into the less obvious but often more telling data that ecommerce affords. This includes leading, often termed “weak” signals about consumer intent that can only be gleaned from the online shopping experience. Think clicks and hovers, not hard transactions.

This new online environment should enable P&G to monitor and respond to shifting consumer trends in real-time, and we are looking forward to seeing how their customer engagement initiative will change the overall online retail landscape. We predict that the insight gained from eStore will improve both indirect and direct sales, as P&G will be able to better collaborate with their distributers on why certain products are selling well, and how to improve upon the merchandising and online sales strategies. If managed well, the P&G eStore should also allow the manufacturer to bring innovative new products to market significantly faster based on real-time visibility into the online browsing behaviors of their customers.

As the title suggests, Forrester believes that information and knowledge management professionals who deal with public websites need to use a mix of different technologies from an ecosystem of providers – no single platform will meet all of their needs.

According to the report’s authors, Stephen Powers, Matthew Brown and Peter Schmidt, the shaky economy has increased pressure on companies to engage customers more efficiently online. We agree that websites offer unmatchable economies of scale when it comes to interacting with customers, enabling better lead generation, increasing engagement, conversions and customer retention rates.

What the report doesn’t say is that the emergence of the real-time Web has also accelerated the need for technologies that can dynamically personalize the online experience and keep pace with rising consumer expectations for instant gratification. We believe this trend will be an even bigger catalyst for investing in these technologies in the months and years to come.

Forrester sees investments rising too:

“Our most recent data suggests this trend will continue into the next year. Consider that in 2010, 51% of organizations plan content management implementations, more than one-third will implement or upgrade customer relationship management tools, and one in four is planning marketing automation software investments.”

While investments in online customer experience technologies are on the rise, Forrester notes that time-to-market for site changes, campaigns, and customer experiences have suffered. According to their client inquiries, “disjointed technology has become a prime factor behind time-to-market issues and the ability to achieve improved online processes.”

If companies can’t rely exclusively on WCM or enterprise marketing suites to support online customer engagement, then what does the bigger ecosystem look like?

Forrester puts several players, including Baynote, into three categories:

Content management vendors include traditional WCM players like SDL, Tridion, Autonomy as well as ECM vendors that offer the fundamentals, plus interactive delivery.

Enterprise marketing vendors include Alterian, Aprimo, Omniture, and Unica, who mainly support campaign management.

New breed engagement vendors, such as Baynote, Backbase and Kapow Technologies, support next-generation Websites.

According to the report, new breed engagement vendors fill important gaps in the overall ecosystem and will be critical pieces of next-generation, more personalized and adaptive websites in the future. We couldn’t agree more.

Collective Intelligence

1)      Tom Austin, Gartner – Tom is a group vice president and Gartner fellow who covers how IT can enhance the performance of individuals, teams and organizations.  His blog includes insightful analysis on the impact that various forms of technology have on productivity. He has been placing a lot of focus on Pattern-Based Strategy, which enables business leaders to actively seek, amplify, examine and exploit new or novel business patterns.

2)      Forrester’s Customer Intelligence Blog – Authored by Forrester analysts Dave Frankland, Julie Katz, and Suresh Vittal, this blog provides details on the latest research and trends affecting marketers that are focused on mining customer data from online behaviors.

3)      MediaPost Behavioral Insider – Steve Smith’s blog takes a fresh look at behavioral marketing, analyzing the latest strategies for marketers to better understand their customers and make strategic decisions that are based on proven approaches.

4)      Destination CRM – This online daily edition of CRM Magazine reports on the latest information on customer relationship management.

Real-Time Web

5)      Steve Gillmor, TechCrunchIT —  Recognized as the leading journalist on the real-time Web, Gillmor has reported on every key player in real-time, either through written blog posts, or in online video episodes with The Gillmor Gang. The Gillmor Gang is a technical show that interviews IT industry executives at major companies that have deep knowledge in the technology that they focus on.

6)      Pete Cashmore, Mashable – The founder and CEO of Mashable, one of the most popular blogs worldwide, Cashmore writes a weekly column for CNN.com as well as regular posts on Mashable. He has been following real-time Web closely lately in his CNN column.

7)      John Borthwick – Currently the CEO of Betaworks, Borthwick’s blog posts study the development of the real-time Web. As an entrepreneur, Borthwick is involved in various companies that touch on the real time Web, and in his own words, is interested in understanding “how media evolves as it collides with real time conversations.”

e-Commerce

8)      Jeffery Roster, Gartner – Jeffery is a research vice president at Gartner as part of the Industry Market Strategies Worldwide unit covering the retail and wholesale industries. His blog posts provide insight into research he is working on, as well as his reflections on industry events including NRF.

9)      Forrester eBusiness & Channel Strategy Professionals Blog – Forrester analyst, Sucharita Mulpuru contributes to this blog every once in a while, and we always find her posts to be very relevant to Baynote. Sucharita publishes the annual Holiday Retail Forecast, and is a recognized authority on technology developments that affect the online commerce industry and vendors that facilitate online marketing and merchandising.

10)   Get Elastic – An e-commerce blog maintained by Elastic Path’s Emerging Media Analyst and eCommerce consultant, Linda Bustos.

11)   Shop.org – An open forum for shop.org members to post their knowledge and experiences with e-commerce.

Search

12)   Gilbane Search Blog – Linda Moulton of analyst and consulting firm, Gilbane Group, blogs regularly about trends and technologies in enterprise search.

13)   Search Engine Watch Blog – This blog by Nathania Johnson provides tips and information about searching the Web, analysis of the search engine industry and help to site owners trying to improve their ability to be found in search engines.

Technology & Innovation

14)   Sramana Mitra on Strategy – An entrepreneur and a strategy consultant in Silicon Valley for over 15 years, Sramana  also writes a weekly column for Forbes and is currently authoring Entrepreneur Journeys, a series of books focused on demystifying entrepreneurship. Her blog posts profile various entrepreneurs, and provides a comprehensive look at the innovation that is happening in Silicon Valley today.

15)   Read/Write Web – One of the world’s top 20 blogs, Richard MacManus, Marshall Kirkpatrick, and their team do an excellent job of analyzing all products that relate to the Internet, and the top trends that impact changes in Internet-related technology. RRW is a great place to remain updated on the latest and greatest Internet innovations.

Customer Service

16)   Ragsdale’s Eye on Service – John Ragsdale is the Vice President of Technology Research for the Technology Services Industry Association (TSIA). He shares the latest in technology innovation, industry events, and his expert commentary on the latest issues effecting customer service executives on his blog.

The excitement reached a fever pitch on Dec. 7 when Google announced it would begin displaying real-time results in its searches.

It’s an electrifying time to be a part of the Web to say the least, and it’s nice to see real-time at the forefront of conversation in the media and at industry events I’m attending.

Given that Baynote is focused on mining real-time implicit behavior on the Web, I’ve been asked lately on numerous occasions what my perspective on the real-time Web is.  Rob Hof, previously of BusinessWeek, also recognized our leadership in real-time back in August, when he featured Baynote as fifth on the list of real-time Web start-ups, behind the likes of Twitter, Facebook and Aardvark.

My take? I believe we are on the cusp of a brand new Web, one that will be driven by the power to tap into both real-time explicit and implicit intelligence, as well as the Collective Intelligence of all Internet users. Although much of the recent excitement about the real-time Web has been focused on gathering and making sense of explicit feedback captured in published sources, such as Twitter, you cannot possibly observe the Web merely by looking at it or reading it. Further, ninety-nine percent of what happens on the Web is never written, never rated, never reviewed. It’s told through implicit behaviors in the form of mouse patterns, clicks and hovers to name a few. It’s told through intent.

In the real-time Web, you can’t just watch what people say, you have to watch what they do.

At Baynote, we are focused on the business use cases of real-time, providing companies with the tools to tap into the implicit behaviors of customers on their sites, and learn, adapt and take action automatically. This Collective Intelligence is reflective of the silent majority, not the loud minority. It is free of bias, and requires no manual processing or analysis.

As the fever pitch for real-time continues to grow, we look forward to helping companies become smarter, more personalized, real-time businesses on the much longer road to actualizing a truly real-time Web. We will continue to watch this trend carefully and plan to write more about the real-time Web for business here in 2010.

Stay tuned for more insight.

Coined by the National Retail Federation a few years ago, Cyber Monday offers retailers a way to organize sales and promotions in an online version of Black Friday, the Friday after Thanksgiving Day. The promising news for retailers is that Cyber Monday hasn’t typically been the largest online shopping day of the year, meaning this season’s peak is likely still yet to come.

I thought readers of this blog may also be interested in my guest article for BusinessWeek, where I share five rules that savvy online retailers already had in place heading into Cyber Monday. For those of you not yet incorporating these practices, fear not. They’re easy to implement and each can help your business year-round.

Over the past few weeks, we’ve explored how there are three key principles that explain why site search just doesn’t perform like we expect it to and what we can consider to help mitigate this.  So far, we’ve learned that:

1. The critical information we need to make search great isn’t in the document – it’s in the users’ heads.
2. Asking users to explicitly provie us information that would improve search, while a seemingly good approach, is inherently flawed.

Today, we explore the third principle that shows that if we want to improve search, we need to focus on all of the things users are doing online.  What I mean here is that we need to look beyond search and at the entire site experience to truly understand what’s valuable, why it’s valuable, and in what context it’s found to be valuable.

Reason #3:  Search does not exist in a vacuum.

In order to improve search, we need to observe more than just search behavior.  Search and navigation have traditionally been seen as two separate paradigms: separate interfaces and separate systems driving them.  But in reality what’s happening?  A user is coming to your site and expressing an interest or intent through their actions.  They might have first expressed that intent through a Yahoo! or Google search that brought them to your site.  They might then express it in the pages they visit and engage with, the navigation they use, the links they click, and maybe the site searches they perform.  This expression of interest may span multiple searches and clicks.  And, finding documents that hold true value for that interest and intent may also take multiple steps.

Let’s think back to the “Insight/Incite” example once more.  Had we only looked at what search results users clicked on, the problem might never have been solved.  Why?  Because the valuable content was never in the results – it wasn’t there to be clicked on in the first place!  To learn what users really meant by “insight”, we had to watch their subsequent navigation, paying particular attention to the patterns of behavior that indicated engagement or that they had discovered content that was of value – even if it happened several steps after the initial search.  Observing search behavior alone is not enough!

What about users who don’t search at all?  What can we learn from them?  Users are actually giving us continual clues to their intent and interest with every link they click and every category they choose.  The documents that users engage with and the order in which they engage also tell us not only about relationships between documents, but intent.  If we take this valuable, implicit insight into account, then we really begin to see how this insight could be used to fix search.

What’s really remarkable is that once we take a step back and think of the entire online experience as a single unified expression of intent and value, we can do a lot more than fix search.  We can start to make recommendations and optimize the user experience with every interaction they make with your site; from the moment they arrive, every step they take through the site, as well as every search they perform. The true goal is to understand the user’s intent and then automatically surface documents that other like-minded peers have found valuable in that same context.  That’s the true wisdom of the crowd, and what Baynote’s Collective Intelligence Platform (CIP) is all about.

There are three key principles that explain why site search just doesn’t perform like we expect it to. Over the next few weeks, I’ll dive deeper into each issue surrounding traditional search and offer my insights and experiences to help you understand why your search sucks, and how you can improve it.

Reason #2:  Actions speak louder than words.

Okay, so we’ve figured out that the critical information is not in the document.  Where is it?  Well, it’s in the users’ heads of course.  Let’s look at an example.  If we look at a work by Shakespeare—or any great work of literature—the meaning cannot be identified simply by looking at the words within it.  It’s synthesized in the reader’s mind, and different readers may derive different meanings based on their own unique makeup and experiences.  The same applies to any document.  We must look beyond the words within the document to truly understand the value.  The key question to ask is this: when and why is this document valuable to users?  Only the users themselves know the answer.

Now that’s all well and good, but how do we extract that knowledge from the users?  We could ask them directly, but while that might seem like a good strategy, it’s actually not.  Asking users to explicitly rank, rate, or tag documents is doomed to failure.  The core problem is one of participation.  Think back to how many times you’ve provided feedback on the web.  Most of us never have; others may have on occasion, but almost certainly not on every page visit or search result.  This participation problem leads to a few key challenges:

A. Low coverage. A small subset of the population rates content, and when they do, the ratings only tend to  cover the most popular content.  Where does that leave the majority of our content?—the long tail.  Unranked and therefore undiscovered.  And with search, it’s not just about knowing that a document has value; it’s about whether it’s valuable for that specific search topic and that user.

Let’s take a deeper look at this.  Let’s say we have a bunch of ratings on a particular camera.  Sounds great, right?  Well, not really.  People like or dislike a camera for a variety of reasons.  Someone looking for a “lightweight camera” might think it stinks, while someone looking for a “cheap camera” might love it.  You can’t ignore the context of what a person is looking for, and getting explicit coverage across every topic of interest is even harder than just getting an overall rating.

And, when you factor in staleness of content — be it the article or the rating — then even the ratings you do have become less meaningful.  Add to that the mountains of content that are being created every day, and the problem really gets out of hand.  There’s just no way to keep up with it if we’re relying on people to go out of their way to explicitly rank, rate, and tag content.

B. Biased. In general, the people who do participate in explicitly rating something online represent a very small subset of the population.  That means that, even for those documents and topics that do have coverage, there’s no guarantee that the knowledge imparted by users even represents the majority opinion.  In fact, it almost always represents fringe opinions that are either extremely positive or negative because those are the people motivated to be heard.

To read more about the bias inherent in explicit methods of capturing community wisdom, check out my “7 Deadly Biases” whitepaper.

C. Inaccurate/Incomplete. Even when an individual decides to provide some form of explicit feedback, it often is not fully representative of even their own experience.  Let’s go back to the insight/incite example from my previous post.  Let’s say that a user decided to tag the Incite product page that was so useful.  What tag do you think they would use?  Probably “incite” or “incite phone,” right?  They would almost certainly not tag it with “insight” once they realized their mistake; but this is actually the tag that would be of most value to the community!  It might surprise you to know that social scientists are generally distrustful of people’s own accounts of their feelings and behavior!  Meta-cognition and emotional self-awareness are far more complex, and less intuitive, than people think.

So, if asking people to tell us what documents are valuable and why they’re of value doesn’t work, then what is the right way?  The key is to observe what people do, not what they say.  It’s both more accurate and more comprehensive.  The wisdom we are looking to tap is present in every single search–whether the searchers were successful or not.  By watching what people do, we can understand which documents are valuable and when.

But we need to be careful here too, because watching which search results users click on is not enough.  Clicks are a very weak indicator: just because people clicked on a result does not mean it’s valuable.  Perhaps the title was intriguing, confusing, or even misleading.  It is critical to follow the user all the way from query to success or ultimate failure—even if several steps later—and not get distracted by what they click on.  At Baynote, we track 24 different behavioral heuristics to ensure that we accurately capture where users are engaging given a particular context and intent.  And it’s not just about search; it’s about the entire online experience.  More on this in the next post.

Next week: Part 4 in the 4-part series, My Search Sucks! where we’ll explore how search does not exist in a vacuum.

My Search Sucks! Part 2

Part 2 in the 4-part series, “My Search Sucks,” discussing why search, well, sucks.

There are three key principles that explain why site search just doesn’t perform like we expect it to. Over the next few weeks, I’ll dive deeper into each issue surrounding traditional search and offer my insights and experiences to help you understand why your search sucks, and how you can improve it.

Reason #1:  The critical information is not in the document.

All full-text search technologies basically work the same way: they look for a match between the words in a user’s query and the words in the text of the documents searched.  That said, there are lots of fancy layers that can be added from simple stemming to complex natural language processing (NLP), but the fundamental assumption is that the engine can figure out which documents best meet a user’s needs by looking inside the document.

While this is a start, it’s just not enough. The critical information isn’t in the document; it’s in someone’s head.  But whose head is it in? Let’s look at some examples.

A favorite example comes to mind involving one of our customers, a large online appliance retailer.  Users were coming to their website looking for a “stove” over and over again, and the search results had “stove-top safe” kettles and pots, but no stoves.  Turns out the reason for this was that this retailer’s website was using the manufacturer terminology, “cooktops” and “ranges.”  The word “stove” was nowhere to be found.  The community was using a different vocabulary than the site.

Sounds like a simple fix, right?  All you need to do is to create a synonym to tell the search engine that a “stove” is the same thing as a “range.”  And sure, once you’ve found and addressed the discrepancy, customers searching for “stoves” will find the “ranges” they’re really looking for.  But what about all of those long-tail terms and content—and what about when things change?

Sam Mefford, an expert in the deployment of enterprise search technologies, commented on last week’s blog.  In his search practice, he sees this challenge surface on a regular basis and provided an example from one of his clients.  The company re-branded one of its products, and made the appropriate changes in its marketing materials and documentation.   Afterwards, field agents and customers could no longer find the products and information they needed, because they continued to search using the old name.  This problem took months to discover.

Another great example is from a customer that’s a well-known wireless provider.  They launched a new LG phone called the “Incite.”  Suddenly, one of the most popular search queries on their site became “insight.”  The search results included lots of business-type documents about how to achieve great “insight” into your business operations, but nothing that matched what users wanted – information on this exciting new phone. Sure, searching for “insight” while the product is called “incite” was technically the user’s mistake, but does that matter when you’re losing opportunities?

Let’s say the words do exist in the document.  It’s often not enough.  There may be 1000s of documents that contain the search terms, but which documents are the best?  A traditional search engine will assume that the one with the most occurrences of the keywords is the most valuable, but this is very often not the case.  Obviously, the technology is more sophisticated than this, but the fundamental basis is along these lines.  The most useful document may only have one instance of the keyword and therefore may be buried on page 10 of the results.  So, how do you get the most useful document to the top of the search results?

Manual tuning is the traditional “solution” to all of these site search issues, but as we discussed earlier, it’s nearly impossible to catch all discrepancies and adapt rapidly—not to mention the time and effort involved.  I’ve even mentioned the spirit of the solution: it’s fundamentally a recognition that the needed information is not in the document, it’s in someone’s head.  But whose head is it really in?

Many companies have experts that manually tune and tweak search.  But that’s a labor-intensive way to temporarily solve the problem and certainly doesn’t guarantee that the expert’s view on what’s right matches with users’.  Why take that chance?  Better to go straight to the source of the information: the user!

Next week: Part 3 in the 4-part series, My Search Sucks! where we’ll explore how actions speak louder than words.

According to Gartner, a pattern-based strategy “provides a framework to proactively seek, model and adapt to leading indicators, often-termed ‘weak’ signals that form patterns in the marketplace.”  For the past several years Baynote has been committed to helping companies identify these patterns with technology that lets them tap into the collective intelligence of customers visiting their websites. This is something that transactional based systems such as business intelligence (BI) and complex event processing (CEP) simply haven’t been able to deliver. Here’s why:

1) For years BI, CEP (more recently) and other related technologies have helped organizations become much more efficient by automating their interactions with customers. However, in the process of creating huge economies of scale, they forced companies to lose the “mom and pop” touch that consumers expect when they walk into a local hardware store or restaurant. In failing to create digital mom and pop experiences, online retailers and publishers have placed unnecessary emphasis on promoting popular products and content, thereby losing out on profits to be gained from merchandising their long tail products.

2) In addition, these so-called “predictive” applications have historically prioritized the wrong set of indicators, often identifying consumer trends weeks, if not months, after the fact. For example, e-commerce transactions lag other more relevant indicators, such as online comparison shopping, by months. Only by tapping into the power of the collective is it possible to see early signals, spot trends and develop strategies around them before your competitors catch on. This holds particularly true for long tail products. Our customer US-Appliance tapped into the implicit behaviors of its website visitors to merchandise colored washers/dryers months before Home Depot and Best Buy began promoting similar products in their stores.

In Gartner’s recent report, entitled “Introducing Pattern-Based Strategy”, they view “the collective” as being critical to developing a pattern-based strategy. We couldn’t agree more with their position:

The collective comprises individuals, groups, communities, mobs, markets and firms that shape the direction of society and business. The collective is not new but technology has made the collective more powerful — and enabled change to happen more rapidly. The explosion of social software has enabled groups and individuals to rapidly form and rally to a cause — often resulting in significant societal changes.

The result for business is a cacophony of rapidly evolving demands, expectations, inputs and transactions, as well as an opportunity to not only react, but to seek signals of change from the collective. Market trends, some subtle, others strong, are masked by noise, and many enterprises are failing to proactively detect the patterns they rely on to direct future strategy and support investment decisions. In addition to failing to detect these patterns, enterprises are not utilizing new resources to proactively seek signals of change nor do they understand their power to influence individuals and communities.

Val Sribar, group vice president of Research at Gartner, sites Amazon’s and Netflix’s use of recommendation engines as good examples of organizations leveraging collective intelligence to support their pattern-based strategies. Sribar agrees with Baynote that recommendation engines identify new patterns in behavior as customers browse and purchase. While Amazon and Netflix are highly popularized cases, we’ve helped hundreds of other well known brands tap into their collective customer networks to significantly increase revenue through cross-selling and upselling, and higher customer loyalty.

We’re excited to see Gartner take a leadership position on this important issue and look forward to working with them and our customers to bring best practices related to collective intelligence to the forefront of modern business strategy.

The report, Americans Reject Tailored Advertising shows that 66 percent of adults said no to tailored ads. Not only that, when informed of specific behavioral targeting techniques that marketers employ to create the ads, even higher percentages — between 73 percent and 86 percent — oppose tailored advertising. Those techniques include tracking behavior on websites and in retail stores.

For a more detailed analysis of the findings, you should check out Stefanie Clifford’s coverage of the report in the New York Times.

We’ve been talking about the pitfalls of behavioral targeting for years, so it’s nice to finally see some national research that tells marketers what consumers actually think about this shady technique. In this age of identity theft and mounting concerns over privacy in general, a practice that proactively profiles a user — perhaps over the scope of many Web sites and over a period of several months — will sound alarms even among the least conservative of us.

Beyond privacy concerns, there are bigger issues with behavioral targeting related to accuracy and quality, that many marketers still don’t understand. Traditional behavioral targeting struggles precisely because it tries to discern what I want now based on my past behaviors. Consider the impact of focusing on historical interests instead of current intent: If I bought a gift for my niece on Amazon.com last week, I certainly don’t want to be bombarded by ads for similar products that probably aren’t relevant during my next visit.

Another way to think of this problem is to consider the idea of roles or what personalization systems might call “profiles”. Humans have far too many roles in life for a profile to possibly predict what a user wants on any given day. A woman shopping for baby clothes, a tie for her husband, and a gift for her sister may appear schizophrenic because she is acting in three different roles — mother, wife and sister. What do you show her next? Tossing strollers ads at her isn’t going to be effective now that she’s shopping for a new cocktail dress for herself.

This is the pitfall of profiles. In a given month, an individual will have thousands of roles. Knowing my past is not necessarily a better way to predict my future. In fact, this phenomenon has been known by psychologists and other scientists for years — humans are animals of context and situations, much less than of historical profiles or roles.

Enter Intent-driven Targeting

An alternative that solves the issues with both privacy and effectiveness is one centered on understanding the users’ intentions, instead of their clickpaths or profiles, and pairing that knowledge with specific content, product and advertising recommendations. This approach relies exclusively on the collective wisdom of like-minded peers who have demonstrated interests or engagement with similar content and contexts.

The concept of profiles is completely removed in this case. Instead, through understanding expressed or implied intent, content appropriate to the user’s current mindset can be delivered.

Most importantly, it kills two birds with one stone: Users get useful, accurate recommendations and ads, while still avoiding the whole privacy mess.