“Nearly three years after the start of the artificial intelligence boom, business technology leaders are starting to change their thinking on return on investment. The new wisdom? Don’t worry so much about AI’s ROI.,” said the WSJ in a recent article, “Stop Worrying About AI’s Return on Investment.” The article is based on discussions with technology leaders at the WSJ Technology Council Summit that took place in New York City on September 15-16. Tech leaders attending the Summit said that “it’s nearly impossible to measure the impact of AI on business productivity. And when we try, we’re measuring it wrong.”
The article cites two main reasons why companies should stop worrying about AI’s ROI at this time. First, “most AI projects are still in the proof-of-concept stage, which is designed to explore what’s possible, not drive a return on investment.” It’s quite possible that few of those projects will make it past that stage, “which is to be expected from a process designed to winnow out all but the best ideas. … One cannot expect significant productivity gains at the pilot level or even at the company unit level. Significant productivity improvements require achieving scale.”
In addition, while AI is still rapidly evolving, CIOs realize that traditional ways of recognizing gains from the technology aren’t cutting it. “Racking up a few minutes of efficiency here and there don’t add up to a meaningful way of measuring ROI. … Only after an AI project scales, or expands across an entire organization, will most corporate technology leaders be able to determine the technology’s true ROI, some experts say.”
Let me discuss each of these key points.
Most AI projects are still in the proof-of-concept stage
In January of 2018 I posted a blog on AI and the Productivity Paradox. I wrote that while AI was already being successfully applied to tasks that not long ago were viewed as the exclusive domain of humans, productivity growth had significantly declined over the previous decade, and income had continued to stagnate for the majority of Americans. This somewhat puzzling contradiction was addressed in “Artificial Intelligences and the Modern Productivity Paradox,”an article by Erik Brynjolfsson, Daniel Rock, and Chad Syverson.
There are no inherent inconsistencies between having both transformative technological advances and lagging productivity, the authors explained. Over the past two centuries we’ve learned that there’s generally a significant time lag between the broad acceptance of new technology-based paradigms and the ensuing economic transformation and institutional recomposition. Even after reaching a tipping point of market acceptance, it takes considerable time, – often decades, – for the new technologies and business models to be widely embraced by companies and industries across the economy, and only then will their benefits follow, — including productivity growth.
We’re in such an in-between period, where multiple overlapping technologies are continuing to emerge from R&D labs into the marketplace, but their full deployment is still ahead of us.
“AI will have implications for the macroeconomy, productivity, wages and inequality, but all of them are very hard to predict,” wrote MIT Nobel laureate economist Daron Acemoglu in “The Simple Macroeconomics of AI,” an article published in May of 2024. “This has not stopped a series of forecasts over the last year, often centering on the productivity gains that AI will trigger. Some experts believe that truly transformative implications, including artificial general intelligence (AGI) enabling AI to perform essentially all human tasks, could be around the corner. Other forecasters are more grounded, but still predict big effects on output.”
According to professor Acemoglu, AI Advances Aren’t Likely to Occur Nearly as Quickly as Many Believe. AI will contribute only modest improvements to worker productivity and will add no more than 1 percent to U.S. economic output over the next decade. Based on his research, he is skeptical of the significantly higher estimates made by AI boosters.
In a subsequent interview with Goldman Sachs senior strategist Allison Nathan, she asked Acemoglu why he was less optimistic on AI’s potential economic impacts than the more bullish predictions of other economists and financial analysts. He replied that the forecast differences are primarily about the timing of AI’s economic impacts rather than about the ultimate promise of the technology. “Generative AI has the potential to fundamentally change the process of scientific discovery, research and development, innovation, new product and material testing, etc. as well as create new products and platforms.” But the economic impact of historically transformative technologies like AI take time to play out.
AI’s true ROI: Productivity or Innovation
The second major reason why companies should stop worrying about AI’s ROI is because at this early stage of AI’a evolution, it’s very hard to measure its true contribution to a business or industry sector, let alone to the overall economy and society. It takes time to determine a technology’s true ROI. “CIOs are recognizing that traditional ways of recognizing gains from the technology aren’t cutting it,” noted the WSJ article.
Tech leaders at the Summit said that “racking up a few minutes of efficiency here and there don’t add up to a meaningful way of measuring ROI. It’s nearly impossible to measure general productivity gains from using AI tools, some event participants said, who preferred to put their efforts on keenly tracking a few critical AI projects, or focusing on innovation.”
“Even in software engineering — where productivity gains are generally considered more straightforward — measurements like the amount of code written by AI don’t necessarily equal a more efficient workforce, tech leaders added. Microsoft’s former chief AI technology officer Sophia Velastegui noted that “For the most impactful business opportunity or products, you have to use AI to really double down on innovation versus productivity.”
Velastegui’s comment brought to mind the remarks by professor Acemoglu in the closing keynote of the 2025 MIT Sloan CIO Symposium, — “The Long Term Evolution of AI Economics: Automation vs Human Complementarity.”
Acemoglu opened the keynote by pointing out that we face two very important choices when considering the longer term evolution of AI, choices that will have major economic implications for the future. One choice is to develop AI tools aimed at improving productivity by automating and replacing human labor. This could reduce the price of many products and services. It would result in lower costs for companies, but “the bottom line for workers would not be that great.” The second choice is to develop AI tools that complement humans and enhance workers’ capabilities, creativity, and productivity.
“The future will have both sorts of technologies,” Acemoglu added. The balance between the two will create different types of winners and losers and “is going to determine the broader impacts of AI.”
From a pure economic point of view, the bias toward automation is understandable. For managers under pressure to reduce costs in a highly competitive environment, automation offers a quick and predictable return. Throughout history, many companies have gone down this path. But, you likely never heard of them “because no company goes into the history books because they have cut costs by 10% or 20%.”
Companies that appear in history books are those that have developed major human-complementary technologies focused on innovation.
In his keynote, Acemoglu reminded us that the Ford Motor Company revolutionized the entire economy by leveraging the growing deployment of electricity in the early 20th century to transform manufacturing with the development of the assembly line. Nobody had previously thought that you could combine a series of electric machines, each with its own small electric motor, to come up with a completely decentralized factory design, which together with much better trained workers, could be used to build different kinds of cars at a fraction of the previous cost, “and create a new market that nobody dreamed of. That was the combination of technology and human resources in a way that was unparalleled for its time.” As a result, the Ford assembly line has long been in history books.
“Instead of worrying about driving ROI off AI pilots, companies should focus on identifying a handful of the most promising AI projects, and make sure that their organizational structures, talent, governance and data infrastructure are up to the task of scaling them,” said the WSJ article in conclusion. “Only after an AI project scales, or expands across an entire organization, will most corporate technology leaders be able to determine the technology’s true ROI.”