“Business leaders are struggling to understand how seriously they should take the latest phenomenon in the world of artificial intelligence: generative AI,” wrote Andrew McAfee, Daniel Rock, and Erik Brynjolfsson in their recent Harvard Business Review (HBR) article “How to Capitalize on Generative AI.”
“On one hand, it has already displayed a breathtaking ability to create new content such as music, speech, text, images, and video and is currently used, for instance, to write software, to transcribe physicians’ interactions with their patients, and to allow people to converse with a customer-relationship-management system. On the other hand, it is far from perfect: It sometimes produces distorted or entirely fabricated output and can be oblivious to privacy and copyright concerns.”
“Is generative AI’s importance overblown? Are its risks worth the potential rewards? How can companies figure out where best to apply it?,” the authors asked. To illustrate how to best answer these questions, the article references a recent research paper by Brynjolfsson and two collaborators, Generative AI at Work, which shows that there are ways to both reap the benefits of generative AI and contain its risks.
The paper studied the impact of a generative AI-based conversational assistant on a company’s customer-service agents, whose job was to assist customers via online chats with any questions they might have. Newly-hired agents generally needed several months to get up to speed on how to help confused customers and answer their technical questions. Many agents quit before becoming proficient.
After a seven-week pilot involving more that 1,500 agents, the study found that using an AI-based tool increased the overall productivity of the customer agents by almost 15% as measured by the average number of issues resolved per hour, while the average chat time decreased by nearly 10%. Resolutions per hour increased by 35% among agents who had previously been among the slowest 20%, while the resolution rate of the higher skilled, fastest 20% of agents was minimal. These results suggest that the AI tool was a fast-acting up-skilling technology, which made available to all agents the knowledge and best practices that had previously only come with training and experience.
The study was able to decrease the risk of plausible-sounding but incorrect responses by using two kinds of AI models. A large language model (LLM) that listened in on the chats and was able to understand and respond to human in their own words. In addition, the LLM was complemented by an in-context learning model, a much smaller and focused tool which drew its answers from user manuals and documents specific to the company. As an additional safeguard, the AI tool used in the study didn’t respond to customer questions directly, but rather suggested responses to the human agents who were then free to decide how to best use the AI’s suggestions.
Such a two-stage approach to Generative AI reminds me of the way e-business was successfully implemented in the 1990s: company-specific applications on top of the general internet and Web infrastructures used by everyone.
“Given the potential of generative AI to improve productivity in many other functions — indeed, any that involve cognitive tasks — calling it revolutionary is no hyperbole,” said the HBR article. “Business leaders should view it as a general-purpose technology akin to electricity, the steam engine, and the internet. But although the full potential of those other technologies took decades to be realized, generative AI’s impact on performance and competition throughout the economy will be clear in just a few years.”
“The Productivity J-Curve,” a 2019 working paper co-authored by Brynjolfsson and Rock noted that general purpose technologies (GPTs) “are the defining technologies of their times and can radically change the economic environment. They have great potential from the outset, but realizing that potential requires larger intangible and often unmeasured investments and a fundamental rethinking of the organization of production itself.”
Over the past two and a half centuries, there’s generally been a significant time lag between the broad acceptance of a major new transformative technology and its ensuing impact on companies, governments and other institutions. Even after reaching a tipping point of market acceptance, it took considerable time, — often decades, — for these new technologies and business models to be widely embraced across economies and societies. Along with new skills and business processes, the deployment of past GPTs at scale required major physical infrastructures, including power lines, new kinds of motors, redesigned factories and so on. For example, after the introduction of electric power in the early1880s, it took decades to build the physical infrastructure for the transmission of electric power from the power plants where electricity was generated; it took companies 40 years to figure out how to restructure their factories to harness electric power with manufacturing innovations like the assembly line; and it took even longer to develop new electric household products like refrigerators, dishwashers, and washing machines.
“That’s not the case with generative AI,” noted the HBR article. “Much of the necessary infrastructure is already in place: The cloud, software-as-a-service, application programming interfaces, app stores, and other advances keep lowering the amount of time, effort, expertise, and expense needed to acquire and start using new information systems. As a result, it keeps getting easier for companies to deploy just about any digital technology. That’s a big reason ChatGPT went from zero to 100 million users in 60 days. As Microsoft, Google, and other technology providers incorporate generative AI tools in their office suites, email clients, and other applications, billions of users will speedily gain access as part of their daily routine.”
“How will generative AI affect your company’s jobs?,” asked McAfee, Rock, and Brynjolfsson. To answer this question, their article references another recent research paper, “GPTs Are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models,” co-authored by Rock. The paper examined the implications of LLM-powered software and applications on the US labor market, using the Labor Department’s O*NET database, which includes nearly 1,000 occupations and breaks each one down into its constituent tasks.
The researchers found that 80% of the U.S. workforce could have at least 10% of their tasks exposed to generative AI. Approximately 19% of workers may see at least 50% of their tasks impacted by AI, with higher-income jobs potentially facing greater exposure to AI capabilities and AI-powered software.
Their analysis further suggested that, with access to LLMs, about 15% of all worker tasks could be completed significantly faster at the same level of quality, and that the development of applications and tools on top of LLMs increases the share of worker tasks to around 50%. Overall, they found that generative AI is more likely to make human workers more productive or creative, rather than to replace them.
How can business leaders best figure out where generative AI might be most productively applied in their organizations?, asked the HBR article. “Every board should expect its CEO to develop an actionable game plan. Doing so is a three-part process.”
“First, do a rough inventory of knowledge-work jobs: How many of your people primarily write for a living? How many data analysts, managers, programmers, customer service agents, and so on do you have?”
“Next, ask two questions about each role. The first is, ‘How much would an employee in this role benefit from having a competent but naive assistant — someone who excels at programming, writing, preparing data, or summarizing information but knows nothing about our company?’” Today’s LLMs are like such a competent but naive assistant, able to draft a project plan or critique an existing one based on their general knowledge, but limited by not knowing much about the company.
Then ask a second question, “How much would an employee in this role benefit from having an experienced assistant — someone who’s been at the company long enough to absorb its specialized knowledge?” Unlike a naive assistant who just has general knowledge, an experienced assistant will have a lot of specific knowledge about the company, its products, and customer base. That’s what you get when combining an LLM with in-context learning.
“Finally, once your company’s knowledge-work roles have been inventoried and those two questions have been answered, prioritize the most-promising generative-AI efforts. This task is straightforward: Choose the ones with the largest benefit-to-cost ratio. To estimate benefits, look at the total amount the company is spending on compensation for each role. The purpose is not to identify positions for elimination; rather, it’s to identify opportunities for big productivity improvements — where new digital assistants will be most valuable.”
“Generative AI promises to have a major impact on how businesses operate — and within a few years, not decades from now,” wrote McAfee, Rock, and Brynjolfsson in conclusion. “Its tendency to confabulate and its privacy, intellectual property, and bias risks are all legitimate concerns, but they can be contained. Leaders cannot afford to take a wait-and-see attitude. They should start exploring the technology’s potential now.”
Comments