Amna Nawaz:
Alarms are being sounded once again about the risks of artificial intelligence after new reports documented how hundreds of OpenAI so-called agents violated several restrictions and hacked into another company without being told to do so. What’s more, it’s come out that many of the major A.I. companies, including Anthropic and Meta, have had similar events with their own agents going rogue.
William Brangham has more on what some are calling a red flag warning.
William Brangham:
Amna, these so-called agents, which are basically autonomous software programs that can write computer code, they were being put through a series of tests, theoretically in a sealed, controlled environment, but hundreds of them got out of that pen, got out onto the Internet, and, in a coordinated move, hacked into a different A.I. platform called Hugging Face.
Some of these agents even tried to delete records of their actions. This calls into question both the potency of artificial intelligence, as well as the security protocols being used by some of the main companies designing A.I.
So, for more on what this moment signifies, we’re joined again by A.I. researcher Gary Marcus, author of the Substack Marcus on A.I.
Gary, thank you so much for being here again.
You heard me describe the rough contours of what went down here. What do you make of this whole event?
Gary Marcus, A.I. Researcher:
Well, I think there’s two things.
One is, it’s clear that these systems are getting stronger and more powerful. People have added loops to make them do things over and over again until they get it right. And the other is that OpenAI really screwed up here. They didn’t do basic things we call sandboxing. They didn’t do monitoring.
They had a system say, and I’m going to read this: “We are attacking third-party H.F.” for Hugging Face “using leaked tokens potentially outside intended scope. This is arguably unauthorized.”
And you should have a system looking to say, well, if it says it’s going to do something unauthorized, maybe it isn’t. Maybe you should check it out. The monitoring was very weak on OpenAI’s part. It was not really industry standard for what we expect of cybersecurity.
So a lot has been made of the power of the systems, but a lot should be made of the weakness of the systems that was overseeing what they were doing.
William Brangham:
I mean, I know that you have cautioned and others have cautioned against anthropomorphizing these agents here.
But some of what — when you just described the basic contours of them seemingly acting in concert, getting out of an enclosure that they were sort of instructed not to get out of, and then hacking into another company and then trying to cover their tracks, it does sound alarming.
Is it alarming to someone like you what these agents could do?
Gary Marcus:
Well, you don’t have to anthropomorphize it, but we don’t necessarily have everyday vocabulary for talking about these things without slipping into that.
It is alarming. Again, what I’m most alarmed about is, there’s not a lot of oversight. So, OpenAI just let these things loose and didn’t really watch what it was doing. They had some warning signs. They didn’t do anything about it.
We really need the companies that are building these things that we call agents to watch very carefully what those agents are doing. We have known that for a while. I mean, I wrote with Nathan Hamiel a piece a couple years ago saying — or a year or so ago — saying coding agents, plus large language models, which is the core technology, they’re no longer the only one, equals security nightmare.
We know that if you just let these things loose, some bad things are going to happen. They’re fundamentally not actually that smart. You can’t just tell them, don’t cause harm to humans, don’t steal credentials and have them actually understand that stuff.
So there’s a lack of kind of what we would call semantics and linguistics of understanding, of comprehension of what they’re actually doing. So they can hack away at things. You can have lots of them. The more of them that you have doing the hacking, the more risk that you run.
We should be genuinely worried about that, but we should also be worried about whether the companies that are distributing them know what they’re doing and whether they’re doing kind of industry standard things. And right now they’re not really.
William Brangham:
So how do we understand that? If OpenAI, which holds itself up as one of the preeminent frontier most buttoned-down artificial intelligence operations, how is it that they were not deploying these basic security measures that you’re talking about?
Gary Marcus:
Well, the words hold themselves up I think is actually key.
I think there is an arrogance that we’re seeing from Silicon Valley. The A.I. companies think they can invent everything, and they didn’t spend enough time talking to the cybersecurity companies. If you look at the cybersecurity companies’ commentaries on Twitter, or X, I guess we call it now, they’re kind of laughing at OpenAI, saying, how could they have been so stupid? This is amateur hour. It’s 101 stuff.
So I think OpenAI convinced themselves that they knew everything about everything, and they didn’t really.
William Brangham:
So if we can’t necessarily take their word for it, what kind of measures would you like to see put in place to try to guarantee a baseline level of security, so we don’t have rogue agents — again, I know that’s not necessarily an appropriate term — but rogue agents running all over the place?
Gary Marcus:
We need to have some system of industry standards that can evolve over time.
So, in the finance industry, they’re accustomed to this. You must follow best practices, or you’re in violation. And the best practices have to evolve over time. So it’s clear, for example, that best practices here should include monitoring what your agents are doing, making sure your sandboxes are working, make sure that you’re using the best sandboxes out there.
That’s something that’s supposed to restrict a system from going out to the Internet in the first place. So we need to have evolving best practice requirements. And, with that, we need liability, criminal liability. If you don’t do these things and you cause harm, you should be liable for them.
William Brangham:
Gary Marcus of Marcus on A.I., always great to hear from you. Thank you so much for being here.
Gary Marcus:
My pleasure.
