The Increasing Influence of Recommendation Systems in Our Everyday Lives

A few years ago, I attended a seminar by University of Toronto professor Avi Goldfarb on the economic value of AI. Goldfarb explained that the best way to assess the impact of a new radical technology is to look at how the technology reduces the cost of a widely used function. Computers, for example, are powerful calculators whose cost of arithmetic and other digital operations have dramatically decreased over the past several decades. As a result, we’ve learned to define all kinds of tasks in terms of digital operations, e.g., financial transactions, inventory management, word processing, photography. Similarly, the Internet and World Wide Web have drastically reduced the cost of communications and of access to all kinds of information, - including numbers, text, pictures, music and videos.

Viewed through this lens, the data and AI revolution can be viewed as reducing the cost of predictions. Predictions mean anticipating what is likely to happen in the future. Over the past decade, increasingly powerful computers, advanced machine learning algorithms, and the explosive growth of big data have enabled us to extract insights from the data and turn them into valuable predictions. As was previously the case with digital operations, communications and access to information, - we’re now able to reframe all kinds of applications as prediction problems. A major such family of applications are recommendation engines or recommender systems, which Wikipedia defines as “a subclass of information filtering system that seeks to predict the ‘rating’ or ‘preference’ a user would give to an item.”

“The essential function of recommender systems is mathematically predicting personal preference,” writes MIT visiting scholar Michael Schrage in his recently published book Recommendation Engines. But, “recommendation engines are far more intriguing and important than their definitions might suggest,” as they not only predict but also shape their users’ preferences. Throughout the book, Schrage explores the human yearning to get good, practical and actionable advice, -  whether from the gods, astrology or self-help books, - and the evolution of recommendations into our ubiquitous algorithm-based technologies to the point where they’ve almost become an extension of our brains. “More people around the world are becoming more reliant - even dependent - upon recommendation engines to better advise, inform, and inspire them.”

“Recommenders increasingly influence how individuals spend their time, money, and energy to get more from life. That explains why global organizations ranging from Alibaba to Netflix to Spotify to Amazon to Google invest so heavily in them.” These powerful platforms are driven by their huge economies of scale, - which are largely based on their highly personalized recommendations. The larger the network, the more data is available for recommending highly personalized offerings. That’s what network effects are all about: the more products or services a platform offers, the more users it will attract, helping it then attract more offerings, which in turn brings in more users, generates more data, and makes the platform even more valuable

Such powerful recommendation engines “not only algorithmically anticipate what ‘people like you’ desire, they nudge users to explore options and opportunities that might never have crossed their minds.” For example, based on personalized recommendations from Netflix, I’ve discovered a number of great films that I’d never have come across on my own, - a major part of the reason I’ve long been a Netflix subscriber. In fact, personalized recommendations have been such a major part of the Netflix brand and business model that, in 2006, the company established the Netflix Prize, an open competition for algorithms that would improve on the algorithms Netflix was using for predicting the user ratings of films based solely on their previous ratings. The competition ended in 2009 with a grand prize award of one million dollars to a team that bested Netflix’s algorithms by 10.06%.

“Recommendation inspires innovation: that serendipitous suggestion - that surprise - not only changes how you see the world, it transforms how you see - and understand - yourself,” argues Schrage. “We have gone from wondering ‘How can people create more valuable innovation?’ to ‘How can innovation create more valuable people?’. That distinction is subtle but profound. The emphasis shifts from innovation as output to innovation as an investment in human capital and capabilities.”

Given their increasing influence in our everyday lives, recommendation systems are poorly understood and their advice is underappreciated. Recommenders add value to their users along four major dimensions: by helping them decide what they could or should do next, such as what route to take to avoid traffic slowdowns; by helping them explore a variety of contextually relevant options, e.g., predicting which search items are the most likely ones we’re actually looking for; by helping them compare those relevant options, such as the costs and user ratings of different makes and models of a product; and, perhaps most critically, recommendation systems help users discover options and opportunities they might not themselves have imagined. “Collectively, this potential help makes recommenders irresistibly appealing to users and developers alike.”

According to Schrage, successful recommendations should be based on five key principles:

• Advice. The advice generated by a recommender system should be “personalized, contextualized and customized to the individual or group it serves,” and it should be presented in the format that’s more likely to be valued.
  

• Awareness. The choices offered should create situational awareness, that is, help people understand the options and opportunities around them. This is especially true when a fairly quick decision is required, such as what route to take to avoid traffic.
  

• Assessment. “How well do the recommendations work? Do people follow the advice? Why or why not? Does following the advice reliably lead to desirable outcomes?” The more aspirational recommenders become, the more they should provide effective assessment tools.

• Accountability. Should recommenders bear any responsibility for bad outcomes or for manipulative advice? The more personal and persuasive the recommender, the more accountability matters.

• Agency. Despite the increasing sophistication of recommendation engines, individuals should retain the power and ability to act independently and exercise choice.

But, their increasing influence on so many aspects of our everyday lives also leads to challenging obligations and expectations. These include:

Trust. “[R]ecommenders enjoy their greatest power, influence, and value when trusted by users. Users confident that recommendations respect their best interest are open to the novel, unexpected, and unproven. They’re not afraid to take a chance. Indeed, they’ll give unknown and untried a shot.”

Privacy. “By design and default, greater personalization requires more personal data and information. Seemingly unrelated datasets may algorithmically blend to yield surprising insights into personal preference. With this innovation trajectory, security and confidentiality become even more important… As with health care, ‘informed consent’ becomes more important as recommenders grow more powerful, pervasive, and predictive.”

Sparsity. Sparsity means a lack of sufficient information. “Even in digital environments with huge numbers of users and items, most users evaluate just a few items. A variety of collaborative filtering and other algorithmic approaches are used to create ‘neighborhoods’ of similarity profiles. But when users rate just a handful of items then ascertaining tastes/preferences - and appropriate recommendation neighborhoods - becomes mathematically challenging.”

Scalability. “As user numbers, items, and options grow, recommendation engines need greater computational horsepower to real-time process data. Determining - with ever-higher resolution and granularity - ‘people like you’ and defining ever-subtler features and attributes of items and experiences for ranking and recommendation are hard problems.”

“With ongoing innovation in machine learning, artificial intelligence, sensors, augmented reality, neural technologies, and other digital media, recommendation’s reach becomes more pervasive, powerful, and important,” writes Schrage in conclusion. “The recommendation future promises to be not just more personal, relevant, and better informed but transformative in ways guaranteed to (persuasively) surprise.”

Comments

[Aaron Brian Silverman]

Irving, I find this interesting, especially in the area of trust, as you mention.

Could I suggest one more criteria which you have written about in the past, which is diversity? Recommendation engines and algorithms need to be developed by a diverse group of developers and data scientists, and the data collected needs to be from a diverse population to help ensure the recommendations will be valuable to all people.

[James Drogan]

Diversity implies a broad view. This seems to me act against the intent of recommendation engines.

The more precise the aim the greater is the likelihood the target will be hit.

Jim

[Mukesh]

The hyper-personalisation means that the recommendation engines will continue to recommend more of the same kind. This is the recipe for us to go into a bubble of sorts in all the areas where we follow the recommendation engines. It feels like we are able to discover new stuff due to the recommendations but in fact, these are not random but along an axis that we are already familiar with.

This is a problem because we are social animals and we live in a society with other humans. We need to be able to gather and discover new stuff to expand our thinking and the inputs (what we consume) play a significant role in the outputs (what we create). So, I believe that as a consumer, we need to consciously build in diversity in our inputs in order for us to be able to create better and make it easier and more interesting to live together.