🔍 Read the full analysis: ChatGPT Made Consultants Better At Data Science—For A While on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. After the chatbot was removed, researchers found no evidence that the ChatGPT group performed better on technical questions without it.
A randomized trial involving consultants at Boston Consulting Group (BCG) found that those given ChatGPT and brief training performed better than a comparison group on coding, statistics and prediction tasks—but researchers found no evidence that the advantage carried over to technical questions answered without the chatbot. The findings, published in Nature Human Behaviour on October 26, 2026, distinguish between doing more with an AI tool and learning to do the work independently.
The trial allocated 986 BCG consultants to a ChatGPT group or a control group that received training in using Google and resources including Stack Overflow and Khan Academy. The main analysis included 487 people who submitted work on both assigned tasks. ChatGPT participants received 15 to 20 minutes of training on prompting and used GPT-4. Each participant completed two 90-minute tasks drawn from coding, statistics and prediction.
Compared with the control group, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The study measured scores against the average of 44 BCG data scientists who completed the tasks without AI. The ChatGPT group still averaged below those data scientists on all three tasks. Researchers also reported that the ChatGPT group finished coding and prediction faster, while the difference in time spent on statistics was not significant.
After the work tasks, participants answered five technical questions without ChatGPT. The authors report that they “find no evidence” the ChatGPT group did better than the control group on this follow-up. They also found that ChatGPT users became more confident that the tool could solve problems it could not, though the paper describes that analysis as exploratory. The trial was preregistered before data collection.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Gains Without Lasting Skill Evidence
The results matter to employers deciding whether generative AI is a productivity aid, a training method, or both. In this trial, ChatGPT helped workers complete technical tasks outside their reported expertise, with the largest relative score difference on coding. But the follow-up test did not show a corresponding improvement in unaided technical performance. Task performance with AI and independent skill are different outcomes, and the study found evidence for the former, not the latter.
That distinction affects how companies interpret output and evaluate training. A stronger deliverable produced with an assistant does not, by itself, show that a worker can reproduce the result without it. For organizations adopting AI, the findings support keeping appropriate review and quality controls in place and measuring unaided learning separately if skill development is a goal. The experiment does not establish that AI never teaches skills; it reports what happened in this particular trial and follow-up.
Products Worth Considering
Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black
Samsung Odyssey OLED G5 27-Inch QHD Gaming Monitor
Apple MacBook Air 13-inch with M5 Chip, 16GB Memory, 512GB SSD, Silver
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The trial took place at BCG in late March and early April 2024, with preregistration dated March 13, 2024. Participants completed two of three tasks: cleaning, merging and filtering datasets with Python; checking a statistical model and interpreting plots and probabilities; or building a model about the predictability of international football matches. The tasks were developed by BCG data scientists with OpenAI’s Economic Impacts Research team.
The researchers designed the tasks so ChatGPT could offer useful guidance but could not correctly answer every question independently. This was intended to test whether participants could use the tool to complete work, rather than simply copy a complete answer. The comparison with 44 BCG data scientists provided an expert benchmark, while the later no-tool questions tested whether any advantage remained when access to ChatGPT ended.
The paper’s title describes the reported effect as “temporary upskilling.” Its authors compare AI assistance to an exoskeleton: a technology that can extend what someone can do while it is in use, without necessarily changing the person’s underlying ability. That framing reflects the trial’s observed gap between assisted task scores and the follow-up results.
““We find no evidence” that the ChatGPT group did better on the technical questions answered without the tool.”
— The study authors
Limits of the Skill Follow-Up
The results come from one employer, a specific set of timed tasks and a trial conducted in 2024 using GPT-4. They do not establish how results would differ in other workplaces, with other AI systems, or after longer training and repeated use. The main analysis also covers 487 participants who completed both tasks, fewer than the 986 initially allocated.
The follow-up consisted of five technical questions, so it provides a limited measure of what participants retained or could do independently. The authors’ finding is that they found no evidence of a group advantage on that test; it is not proof that no participant learned anything. The reported increase in confidence about ChatGPT’s abilities is described by the authors as exploratory, rather than a definitive conclusion about all users.
What Employers Should Measure
The study does not announce a new trial or a scheduled follow-up. For employers, the practical next step is to distinguish AI-assisted output from unaided capability when evaluating training and work quality. Organizations seeking to know whether employees have learned a task would need to assess performance both with the tool and without it, while accounting for the limits of those assessments.
Further research would be needed to determine whether longer or different forms of instruction can produce skills that persist after AI access ends, and whether results transfer to other jobs and tools. Until then, this trial supports a measured conclusion: ChatGPT improved performance on the tested tasks, while the study did not detect a matching advantage on its short no-tool test.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. They still averaged below the BCG data scientists who completed the tasks without AI.
Did the consultants retain the advantage without ChatGPT?
Researchers found no evidence of a group advantage when participants later answered five technical questions without ChatGPT. The authors caution through their study design that this follow-up is a specific test, not a complete measure of everything participants may have learned.
How was the study conducted?
Consultants were assigned to ChatGPT training or training in Google and online resources, then completed two timed data science tasks. The trial was conducted at BCG in March and April 2024 and published in Nature Human Behaviour in October 2026.
Does the study show that ChatGPT cannot teach skills?
No. It reports that this trial found improved task performance with ChatGPT but no detectable advantage on a short test without the tool. It does not settle whether different training, longer practice or other uses of AI can lead to lasting skill gains.
Source: ThorstenMeyerAI.com
