Fear not, AI can actually replace our jobs in the next decades or so, but as a whole, humans still have the authority to use them.
OpenAI cofounder and former Tesla AI lead Andrej Karpathy says that we should hit the brakes. In a talk at a Y Combinator event this week, Karpathy called on developers and leaders in tech to “keep AI on a leash,” cautioning that large language models (LLMs), though impressive, are still fundamentally flawed and unreliable without human oversight.
The Problem with Overreliance on AI Agents
OpenAI’s Andrej Karpathy Warns Against Unleashing Unsupervised Agents Too Soon
Levart_Photographer/Unsplash
Even as AI is capable of producing thousands of lines of code in mere seconds, Andrej Karpathy said that this does not eliminate the developer’s position; it just transforms it.
“I’m still the bottleneck,” he explained, adding that the developers need to meticulously check the output so that they do not introduce severe bugs. It’s not only about incorrect code but also about the unpredictability of LLMs.
Karpathy likened LLMs to “people spirits,” weird copycat versions of human smarts that hallucinate data, lose context, and produce absurd mistakes. “
He continued that AI can still be left behind in simple logic and facts when it claims that 0.11 is larger than 9.9, or if there are two R’s in strawberry.
Read more:
AI Cheat Scandal: Blue Books Make a Comeback as Schools Fight Back Against ChatGPT Takeover
Incremental Prompting and Cautious Engineering
To avoid these dangers, Karpathy suggests a gradual, systematic process in dealing with AI—particularly when coding. He suggests employing “small incremental chunks” and designing tangible, precise requests.
Business Insider reported that Karpathy came up with the term vibe coding earlier this year to characterize a loose, prompt-laden paradigm of AI coding, but now he’s definitely signaling that although the vibes are great, discipline is paramount.
Supervision Is Imperative
Karpathy is hardly alone in his reserved position. Bob McGrew, a former research lead at OpenAI, recently voiced similar views on Sequoia Capital’s “Training Data” podcast.
Human engineers, as McGrew sees it, are still essential not only for monitoring, but for deconstructing complicated tasks that AI cannot fully comprehend or handle yet.
Meanwhile, software engineering legend Kent Beck, co-author of the Agile Manifesto, offered a vivid metaphor: AI agents are like genies.
Genies can grant your wish, but not in the most desirable way you pictured they would arrive. Beck noted that AI often misunderstands the intent behind instructions, leading to unpredictable or counterproductive results. This makes the entire process feel “like gambling.”
Big Tech Still Betting on AI for Coding
Amid these worries, the top tech firms continue to drive AI further into software engineering. Alphabet CEO Sundar Pichai announced that more than 30% of the firm’s newly written code now originates from AI, a rise from 25% previously.
Sundar also preached earlier this month that AI is here to generate jobs and not kill them.
Nevertheless, as Karpathy and other experts warn, AI use at scale in coding has to be accompanied by careful regulation. Without it, even the most intelligent agents can run amok, and coders might be left with a very costly mess to clean up.
Related Article:
Powerful Language Models Are Supercharging Carbon Emissions, New Study Warns: What’s the Solution?