Tech research
What AI really
does at work.
The best experiments so far tell a stranger story than the hype. Beginners gain the most, experts can slow down without noticing, and a flood of AI output barely reaches the finish line.
Ask anyone who uses a chatbot at work whether it makes them faster, and most will say yes without a second thought. Researchers have spent the past few years checking, with stopwatches, control groups and payroll records instead of opinions. Their answer is more interesting than yes.
Take the trial run by METR, a nonprofit research lab. Sixteen experienced software developers worked through 246 real tasks on projects they knew inside out, and a coin flip decided whether each task could use AI. Before they started, the developers predicted AI would cut their time by 24%. After they finished, they still believed it had saved them 20%. The clock said the opposite: with AI allowed, the work took 19% longer.
That gap between how AI feels and what it delivers runs through the best research on AI at work. The gains are real. They are also uneven, easy to overestimate and surprisingly easy to lose before they reach the bottom line. Here is what the strongest studies found, and what it means if you run a business.
The biggest winners are the newest people
The largest study of AI in a real workplace followed 5,172 customer support agents as their company rolled out an assistant that suggested replies while they chatted with customers. Economists Erik Brynjolfsson, Danielle Li and Lindsey Raymond found that agents with the assistant resolved 15% more issues per hour on average.
The average hides the real story. The least experienced and least skilled agents gained about 30%. Agents two months into the job with AI performed as well as agents with more than six months of experience without it. The most experienced agents barely sped up, and the quality of their work dipped slightly.
The explanation is simple, and a little profound. The assistant had learned from the conversations of the best agents, so it quietly passed their habits on to everyone else. Customers noticed too. They were more polite, and less likely to ask for a manager.
Brilliant, until it is not
In a study run with Boston Consulting Group, researchers from Harvard Business School, Wharton and MIT gave 758 consultants realistic client work and randomly chose who could use a leading AI model. On 18 tasks well inside what the AI could do, from creative work to analysis, the consultants who used it finished 12.2% more tasks, worked 25.1% faster and turned in clearly better work.
Then came a task picked to sit just beyond the AI’s abilities: a tricky business problem that looked no harder than the rest. There, consultants using AI were 19% less likely to reach the right answer than those working alone. The researchers call this the jagged frontier. AI is excellent at some tasks and misleading at others that look just as hard, and nothing on the surface tells you which is which.
The METR developers ran into the same edge, with a twist. They were experts on their own projects, exactly the people the support study says gain least, and they did not feel the slowdown at all. When METR tried to repeat the test with newer tools, it hit a telling problem: 30% to 50% of the developers said they were holding back some tasks because they did not want to do them without AI. METR now thinks newer tools probably do speed developers up, but it could no longer measure by how much.
More activity is not more finished work
The newest and largest study of AI and output looks at software, where every step leaves a trace. Mert Demirer, Leon Musolff and Liyuan Yang of MIT and Wharton tracked more than 500,000 developers on GitHub, along with data on how each of them used AI.
Every generation of tools raised coding activity further. Autocomplete lifted it by 30%. Interactive coding agents took the total gain to 180%. Autonomous agents took it to 240%. Then look at what shipped. That 240% jump in activity became an 80% rise in projects and only a 30% rise in releases. Across four major app marketplaces, the number of new apps rose sharply, yet total use of new apps did not rise at all.
The authors call it the weak link. AI speeds up the parts of a job it touches, but the work still has to be reviewed, approved, fixed and released by people, and those steps become the bottleneck. More drafts are not more finished work.
A randomized trial inside Alibaba’s customer service shows the same pattern with customers watching. An AI agent was switched on for about half of 647 service workers, across roughly 680,000 chats. Chats the agent finished alone were far quicker, but customers rated them almost a full point lower on a five point scale. The worst case came when the system waited until a customer was visibly frustrated before calling in a person. Those chats ran longer, more customers came back with the same problem within a week, and ratings fell by nearly a full point. When supervisors stepped in early, on their own, the drop in ratings shrank to about half a point.
Paychecks have not moved. Hiring might have.
If AI made people dramatically more productive, you would expect it to show up in pay and hours. Anders Humlum and Emilie Vestergaard of Chicago Booth and the University of Copenhagen linked large surveys to Denmark’s official records of earnings and hours to find out. They describe rapid currents under still water. Most employers in the exposed occupations had launched chatbot initiatives, workers reported real benefits, and new AI tasks were everywhere. Yet they found no measurable effect on earnings or recorded hours, precise enough to rule out anything larger than 2%. What changed was the shape of the work, with new tasks such as writing with AI, checking its output and fitting it into the job.
The sharpest signal so far comes from the United States. Using payroll records from ADP, Erik Brynjolfsson, Bharat Chandar and Ruyu Chen of Stanford found no sign of widespread job losses. But employment of workers aged 22 to 25 in the jobs most exposed to AI now stands 19% below where it would be had it kept pace with their peers in less exposed jobs. The gap comes mostly from fewer young people being hired rather than more being let go, and it is concentrated in jobs where AI replaces tasks rather than helps with them. The authors are careful to call it a pattern, not proof.
What it means for your business
The research does not say AI is overhyped, and it does not say AI will save you. It says the payoff depends on where you point it and how you check it.
- Start with your newest people and your most repeatable work. That is where the gains were largest, and where AI can pass on what your best people already know.
- Measure, do not poll. The developers who felt 20% faster were 19% slower. Track time per job, rework and customer ratings before and after you roll anything out.
- Find your own jagged edge. Try AI on real tasks from your business and check the answers. Keep people in charge of judgment calls, pricing and anything unusual.
- If AI talks to your customers, hand over early. A person who steps in before the customer gets annoyed protects your ratings. A rescue after the fact is the worst outcome of all.
- Plan for the new bottleneck. AI multiplies drafts far faster than it multiplies finished work. Your capacity to check, approve and publish decides whether you see the gains.
- Keep hiring and training juniors. AI makes new people productive sooner, and the clearest warning in the data is that companies are hiring fewer of them.
The same logic holds in marketing. AI can draft ten ads in a minute, but it takes people who know your market to pick the one that sells, and to prove it worked. If you want help putting AI to work where it pays, book a strategy call.
The research
Every number in this story comes from one of these sources. Working papers, reports and industry data are marked, because other researchers have not reviewed them.
- An AI assistant for 5,172 customer support agents By Erik Brynjolfsson, Danielle Li and Lindsey Raymond Read the study (opens in a new tab)
- The jagged frontier: 758 consultants at Boston Consulting Group By Fabrizio Dell'Acqua, Edward McFowland III and colleagues Read the study (opens in a new tab)
- Sixteen experienced developers, with and without AI By Joel Becker, Nate Rush and colleagues Read the study (opens in a new tab)
- The same test, tried again with newer tools By Joel Becker, Nate Rush and colleagues Read the study (opens in a new tab)
- Writing code versus shipping code, across 500,000 developers By Mert Demirer, Leon Musolff and Liyuan Yang Read the study (opens in a new tab)
- An AI agent in Alibaba's customer service By Yiwei Wang, Chuan Zhu and colleagues Read the study (opens in a new tab)
- Chatbots and paychecks in Denmark By Anders Humlum and Emilie Vestergaard Read the study (opens in a new tab)
- Young workers in the jobs most exposed to AI By Erik Brynjolfsson, Bharat Chandar and Ruyu Chen Read the study (opens in a new tab)