“The emergence of generative artificial intelligence (AI) has attracted significant attention for its potential economic impact,” wrote economists Erik Brynjolfsson, Danielle Li, and Lindsey Raymond in a recently published article, “Generative AI at Work.” “Although various generative AI tools have performed well in laboratory settings, questions remain about their effectiveness in the real world, where they may encounter unfamiliar problems, face organizational resistance, or provide misleading information.”
Their article studied the impact of generative AI (GenAI) on productivity and worker experience in the customer-service sector, an industry with one of the highest rates of AI adoption. It analyzed survey data from 5,172 customer-support agents working for a Fortune 500 firm that sells business-process software in order to quantify the effect of deploying GenAI at scale in the workplace. The AI system used in the study was designed to monitor customer-service chats and provide the agents with real-time suggestions for how to respond. While the AI system is intended to assist the agents, the agents remain responsible for the overall conversation and are free to ignore or edit the AI's suggestions.
“Computers and software have transformed the economy with their ability to perform certain tasks with far more precision, speed, and consistency than humans,” said the authors. “To be effective, these systems typically require explicit and detailed instructions for how to transform inputs into outputs: a software engineer must program computers. Despite significant advances in traditional computing, many workplace activities, such as writing emails, analyzing data, or creating presentations, are difficult to codify and have therefore defied computerization.”
The new generation of data-centric AI systems work differently from traditional computer software. These systems, — which can be viewed as a kind of software 2.0, have two major components: the models used to make predictions and the data used to train the models to enable them to make predictions. The centrality of data is the common element in the key technologies that have advanced AI over the past 20-25 years, including big data and advanced analytics in the 2000s, machine and deep learning in the 2010s, and more recently foundation models, LLMs, and generative AI, with agentic AI now poised to take AI to the next level. Data is viewed not merely as fuel for AI, but as a determining factor in the overall system quality, and a way to help build AI systems that deal with complex real-world problems.
After analyzing the agents’ customer-support data, the study came up with four sets of findings.
First: AI assistance increases worker productivity, resulting in a 15% increase in the number of chats that an agent successfully resolves per hour. These productivity increases are based on shifts in three components of overall productivity:
- a decline in the time it takes an agent to handle an individual chat;
- an increase in the number of chats that an agent handles per hour; and
- a small increase in the share of chats that are successfully resolved.
Second: the impact of AI assistance varies widely among agents:
- Less skilled and less experienced workers improve significantly across all productivity measures, including a 30% increase in the number of issues resolved per hour;
- AI helps newer agents move more quickly down the experience curve: treated agents with two months of tenure perform just as well as untreated agents with more than six months of tenure;
- AI has little effect on the productivity of higher-skilled or more experienced workers; and
- AI assistance leads to a small decrease in the quality of conversations conducted by the most skilled agents.
Third: a number of mechanisms underlie the study’s main findings:
- Agents who follow AI recommendations more closely see larger gains in productivity over time;
- Experience with AI recommendations can lead to durable learning;
- Workers see productivity gains relative to their pre-AI baseline even when the AI recommendations are no longer available.
- Gains are most pronounced among workers who had greater exposure to AI and followed the AI suggestions more closely.
- Gains from AI adoption are largest for relatively rare problems, perhaps because agents are already capable of addressing the problems they encounter most frequently;
- The AI system may not provide any suggestions when there is insufficient training data for a topic;
- Access to AI improves the English-language fluency, especially among international agents; and
- AI adoption enable low-skill agents to begin communicating more like high-skill agents.
Fourth: contact-center work is often challenging, with agents frequently facing hostile interactions from anonymous, frustrated customers. AI assistance not only helps agents communicate more effectively, but it also has an impact on the overall customer-agent interaction:
- The findings show that access to AI assistance significantly improves the customer treatment of agents, as reflected in the tone of customer messages;
- Customers are less likely to question agents’ competence by asking to speak to a supervisor; and
- These changes come alongside a decrease in worker attrition, which is driven by the retention of newer workers.
“Our findings show that access to generative AI suggestions can increase the productivity of individual workers and improve their experience of work.” wrote the authors. “In the longer run, firms may respond to increasing productivity among novice workers by hiring more of them or by seeking to develop more powerful AI systems that replace labor altogether. While the introduction of generative AI may increase demand for lower-skill labor within an occupation, the equilibrium response to AI assistance may lead to across-occupation shifts in labor demand that instead benefit higher-skill workers.” The survey data analyzed in this study is insufficient to observe changes in wages, overall labor demand, or the skill composition of the workers hired at the firm.
“The results also underscore the longer-term challenges raised by the adoption of AI systems. In our data, top workers increase their adherence to AI recommendations, even though those recommendations marginally decrease the quality of their conversations. Yet with fewer original contributions from the most skilled workers, future iterations of the AI model may be less effective in solving new problems. Our work therefore raises questions about how these dynamics play out over the long run.”
Let me summarize a few of the paper’s key conclusions.
“In our setting, we find that access to AI-generated recommendations increases overall worker productivity by 15%, with even larger effects for lower-skill and novice agents. These productivity gains in part reflect durable worker learning rather than rote reliance on AI suggestions. Furthermore, AI assistance appears to improve worker on-the-job experiences, such as by improving customer sentiment and confidence, and is associated with reductions in turnover.”
However, the current study raises many unanswered questions:
- The findings only apply to a particular AI tool used in a single firm and a single occupation and should not be generalized across all occupations, firms, and AI systems;
- The setting for this study involves a relatively stable product and set of technical support questions;
- In a setting where the products or environment is changing rapidly, the findings and recommendation may well be different; and
- More generally, we are not able to observe longer-run equilibrium response
“In principle, the increased productivity we observe could lead to either lower or higher demand for customer-service agents.” If the productivity gains enables the firm to handle the same number of customer service issues with fewer worker-hours, it would lead to less demand for human labor. Conversely, if AI assistance improves the overall customer service experience, — such as shorter wait times and higher-quality service, — it may significantly increase the demand for product support and human labor.
“Finally, our findings also raise questions about the nature of worker productivity. Traditionally, a support agent's productivity refers to their ability to help the customers. Yet in a setting where customer-service conversations are fed into training data sets, a worker's productivity also includes the AI training data they produce. Top performers, in particular, contribute many of the examples used to train the AI system we study. This increases their value to the firm. At the same time, our results suggest that access to AI suggestions may lead them to put less effort into coming up with new solutions. Going forward, compensation policies that provide incentives for people to contribute to model training could be important. Given the early stage of generative AI, these and other questions deserve further scrutiny.”
Comments