Here is how the story usually goes. You have been paying a vendor to provide product recommendations for your website. Someone from the technology team says that they can develop that technology internally. Their project will not only save money, but will be more innovative than the vendor. It’s a win-win!
The truth is simple product-to-product recommenders are conceptually easy to build. The collaborative filtering techniques like “people who viewed this also viewed these items” and “people who bought this also bought these items” are well known. Software, like Hadoop, and even the algorithms are available in open source. These basic techniques are even taught in school where students are given data sets and then asked to create predictions of the products that should be shown together. Your tech lead and developers are excited to work with Hadoop and other big data stuff. So, why not put together a small team and knock this project out in a sprint or two! Here are three simple reasons you should think twice about building your own recommender system.
Pitfall 1: True Development Cost –You should be able to get a project up and running in a couple months, right? Unfortunately, using open source technology and making the technology scale in a production environment are completely different problems. Just getting your hands on the data you will need to get started is a project. Resources you’ll need to ensure the project is a success – (1) a data scientist to understand the algorithms, (2) a data engineer and some developers to help implement the technology, and (3) some developers who specialize in performance, scalability and reliability. In one very public example of an in-house recommender system, the company had one data scientist and six developers on the project for two years before the project was ready to be launched. You can do the math.
Pitfall 2: Don’t Forget about Maintenance –The kinds of developers who want to work on the next cool project are not the same ones who you can task with product maintenance. So, the team that builds your new product recommender technology is going to move on to another project. Most likely, nobody is going to work on the recommender project –until it breaks. In one example that I know of, the business team had no idea who had originally developed the technology and no way to update it.
Pitfall 3 – It’s Old News – One key reasons for building in-house technology is to gain competitive advantage. Internally developed product recommenders are often billed as innovation initiatives. The reality is far from innovative. The project most retailers can develop is going to be based on five year old technology, using ten year old techniques. A project using algorithms that everyone else has access to will at best create parity and not advantage. Besides, the personalization engines available today have moved far beyond simple product recommenders to using massive volumes of customer data, real-time processing and advanced algorithms to create one-to-one personalized experiences at scale.
While building a recommender system might seem like a fun and innovative project, this is clearly one area where it still makes sense to partner. Personalization engines have the advantage of specialization and focus, multiple clients, more data, more resources to invest, and production systems to run on. When you look at the costs and benefits to developing an in-house recommender system, you can clearly see that you will likely spend considerable resources but you will not have bought any advantage. Instead, you will have wasted time and resources that you could have spent developing new products or competitive advantage in other areas.