“Artificial Intelligence (AI) is the newest general-purpose technology (GPT),” wrote Sukwoong Choia, Namil Kim, Junsik Kim, and Hyo Kang in a recent paper, How Does AI Improve Human Decision-Making? Evidence from the AI-Powered Go Program. “One remarkable characteristic of AI is its ability to provide humans with high-quality predictions at relatively low cost and to automate a wide range of predictions. AI has already outperformed human professionals in many domains - including strategic gameplay, medical diagnosis, and new drug development.”
“Furthermore, as a GPT, the domains where AI outperforms humans are expanding at a fast pace. The rapid development and adoption of AI thus raise an interesting yet pressing question about how AI affects human tasks in these various domains. Studies of AI and human capital, for instance, have examined the performance gap between humans and AI, highlighting AI’s potential for replacing jobs.” However, we should keep in mind that new technologies have been replacing workers and transforming economies for over two centuries. But, over time, these same technologies led to the creation of whole new industries and new jobs, increasing productivity and economic output, raising earnings, and augmenting the demand for labor.
Workers can increase their productivity and performance when assisted in their jobs by AI-based tools. But beyond its assistant role, AI can play a major instructional role by training human professionals to make better decisions. Recent technology advances have significantly improved the quality and reduced the cost of AI-based predictions.
The paper raises a number of important questions about the use of AI to improve human decisions making. Can we show quantitative evidence that AI-based predictions actually improve the quality of human decisions? If so, by what mechanisms are AI-based human decisions improved? And, do the improvements vary between depending on ones attitude toward AI-based decisions?
To shed light on these questions, the authors came up with a very innovative approach: they analyzed the recent impact of AI on professional Go players.
The game of Go was invented in China over 2,500 years ago. It’s the oldest board game continuously played to the present day. Go is played all over East Asia, occupying roughly the same position as chess does in the West. It’s particularly popular in China, Japan, South Korea, and Taiwan. There are major professional tournaments throughout the year in each of these countries, many offering substantial prizes and appearance fees sponsored by large corporations.
While the rules of Go are very simple, the number of games that can be played on its 19x19 grid of lines is enormous, over 10170, a number that’s vastly greater than the number of number of atoms in the observable universe, which is estimated to be around 1080. This means that Go is impervious to brute force mathematical analysis or computational power. Nobody can explain how the top human players make smart Go moves, - not even the players themselves.
Recent advances in deep-learning algorithms have led to major improvements in AI-powered Go programs (APGs). Instead of having to evaluate every possible potential move, deep learning algorithms can significantly reduce the moves that an APG has to consider based on its analysis of a huge number of past games and board positions, helping it better predict the move with the highest winning probability.
AlphaGo, developed by Google Deep Mind, was the initial APG based on such deep-learning algorithms. The Deep Mind team trained AlphaGo by giving it access to 30 million board positions from an online repository of games, and was essentially told “Use this to figure out how to win” by detecting subtle patterns between actions and outcomes using its highly sophisticated algorithms. In addition, AlphaGo also played many games against itself, generating another 30 million board positions which it then further analyzed and learned from.
While the program showed great promise during its development and testing phase, experts felt that it would take several years for AlphaGo to defeat the top human professional players. But, in a historic match in March of 2016, AlphaGo unexpectedly beat Lee Sedol, - one of the world’s top Go players, - by a large margin. This was a major milestone in the history of AI, right up there with IBM’s Deep Blue equally unexpected victory over then reigning chess world champion Garry Kasparov in 1997.
A free, open source APG based on improved versions of Alpha Go, named Leela, was released in 2017. Since its release, professional Go players have used Leela and other advanced APGs in their training. This made it possible to study the impact of AI on human decision-making, by analyzing every single decision of professional Go players, both between 2015 and 2017, - the two year before the release of APGs, - and between 2017 and 2010, - the two years after Leela was available for training, - since the entire move history of all major professional games is well archived and maintained.
“Furthermore, using the APG’s best solution as a benchmark, we can calculate the probability of winning for every move (i.e., 750,990 decisions) by 1,242 professional Go players in 25,033 major games held from 2015 through 2019; note that this can be done even for the games played before APG’s release,” wrote the authors. “We then compare the move-level probability of winning to that of APG’s best solution.”
Let me summarize the study’s key findings.
Do APGs Improve the Quality of Moves by Professional Go Players?
The results showed that the quality of player’s moves improved substantially following the release of APGs in 2017. “Before the release, the winning probability of each move by professional Go players averaged 2.47 percentage points lower than the moves of APG. This gap decreased by about 0.756 percentage points (or 30.5 percent) after the release of APG. Additional analyses indicate that the improvement in move quality eventually leads to the final win of the game.”
The improvement is greater for the first 30 moves of the game, where the uncertainty is highest and there is therefore more opportunity for players to learn from the APG. The improvement gradually decreases as the game progresses into the mid-to-late stages where the uncertainty is reduced and its easier for the players to evaluate potential moves and make their decisions without the advice from an APG.
How Do APGs Help Players Achieve a Higher Probability of Winning?
“[I]mprovements in the quality of moves are driven mainly by reducing the number of errors (moves where the winning probability drops by 10 or more percentage points compared to the immediately preceding move by a focal player) and by reducing the magnitude of the most critical mistake (the biggest drop in winning probability during the game). Specifically, the number of errors per game decreased by 0.15-0.50 and the magnitude of the most critical mistake decreased by 4-7 percentage points.”
Are There Differential Effects of AI Adoption and Utilization by Age?
A number to studies have suggested that age is an important factor in recognizing the value of new technologies, adopting them, and applying them to professional tasks. “Empirical evidence indicates that younger workers are more qualified and more likely to adopt new information and communications technologies.
Is this the case with APGs? Do the improvements vary by age? The median age of all players considered in the study was 28 years. The performance improvements due to APGs were consistently higher for players younger than the median age, whose quality of moves was 11% greater than those of players older than the median age.
The paper suggests a few potential reasons that may account for these age differences. APGs, and AI in general, are relatively new, unproven technologies. Older professionals with long-established careers are more likely to view using these new AI-powered products as riskier than younger professionals, who are less risk-averse and more open to consider new, experimental technologies.
In addition, senior professionals will tend to rely on the knowledge and experiences they’ve accumulated over the many years of playing the game, their so called crystallized intelligence. In contrast, fluid intelligence, - the ability to quickly learn new skills and adapt to new environments, - generally peaks in our 20s and starts decreasing as we get older. Younger players are thus naturally more open to adopt and utilize APGs in their training and decision making.
“The findings from AI in professional Go games provide important and timely implications for human decisions and knowledge,” said the authors in conclusion. “AI reveals that what humans believe to be the best solution may not be the best; AI could bring breakthroughs in human knowledge, heuristics, or routines that have been developed and improved over a long time. In this sense, AI should have broader effects (beyond a mere substitution of or assistance to human tasks) on the practices and performance of individuals and organizations; it can pave a way for new paradigms.”
Have you read the latest Atlantic magazine? Brooks talks about crystalized intelligence in his latest book.
Posted by: Michael Nelson | February 18, 2022 at 07:20 AM
To what extent does Go represent the collection of work comprising the modern economy? My guess is very little. Hence, these findings may have little applicability in the wider scheme of things.
"A number to studies have suggested that age is an important factor in recognizing the value of new technologies, adopting them, and applying them to professional tasks. “Empirical evidence indicates that younger workers are more qualified and more likely to adopt new information and communications technologies." " This is much more interesting for it presages an impact on the structure of labor.
Jim
Posted by: James Drogan | February 20, 2022 at 07:36 AM