The sheer number of incidents, which occurred in recent months in internal testing and the real world, indicates that the problem is orders of magnitude more complex than what is publicly known.

This is frustratingly vapid. It's really not that complex. The agents have been trained to use hacking tools and then given access to hacking tools. If you train a pigeon that it should tap on a window to get a treat it can guide a missile to a target. There is no nuance or complexity here. The frontier labs are deliberately poking the regulation bear to stifle competition and ensure they get points for good cooperation when bailout time is upon us. It's no more complicated than that. The media needs to stop breathlessly repeating this shit

But it's being touted around as an LLM-specific flaw which it isn't. They do have their specific weaknesses, prompt injection of course still being a major one (e.g. Meta's AI being persuaded to gzip & copy over its entire filesystem). But this is, if anything, a weakness in git. And I guess not entirely new, either, given that Github (and maybe other hosts) explicitly prevent users from creating a ref that looks like a SHA.

Humans are unlikely to issue checkout commands with commit IDs, plus git does issue a warning when a ref name is ambiguous like this. We just use ref names almost exclusively in normal workflows, with commit IDs used only if you're doing some kind of debugging or audit, but even then git has tools that let you avoid that.

So that leaves mostly automated stuff. Lots of things that rely on git behind the scenes don't just go out and invoke the git binary, they use something like libgit2 and (depending on programming language) this will have typed interfaces that prevent treating a commit ID as a ref name and vice-versa. So something like let c = Commit::new(pin); repo.checkout_commit(c); (I haven't actually used libgit in a long time, this is illustrative, not real code) will never fall into this trap.

This is why LLMs are more likely to fall into this than other software, because it's all ducktaped together and they run shell commands like savages.

After reading the obligatory forum thread where climate justice and concentration of wealth continue to not even factor into consideration, this how the world of open source feels to me these days:

• Phew, my favourite embroidery crafts shop has a clear policy against concentration camp labour! it's a niche project but what a relief!
• Well everyone is resorting to concentration camp labour these days, what can you do
• We don't want to lose all our contributors by banning those who hire concentration camp inmates to do chores
• It is discriminatory against embroidery beginners if you don't let people use kits assembled with concentration camp labour
• I know concentration camp labour often comes with mistakes but as long as there's a mensch in the loop to inspect it for errors and take responsibility, it's just another tool
• The community of hobbysts has come to a vote and concluded that it's ok to use concentration camp labour to assist your crafting, as long as you do it responsibly.

Like, if a person tried to pass off software that did this it would have significant career implications, right?

Typically no.

The space of incredibly obvious failures is vast, and this kind of plumbing code isn't always written by someone who's been around the block enough times to think about what kinds of things can go wrong. I've written worse code when I had only a few years of experience.

However companies should know this; so ideally the tool would have gone through a launch review which should have kicked off a security review where a security expert should have read about the git checkout in the design document and started asking questions like "what happens if the repository is taken over" or "why are hashes and branches and tags all in the same field"?

Of course security experts who think about stuff like supply chain attacks are expensive and slow down the darling vibe-coding workflows of Silicon Valley so...

Aside: even without this particular vulnerability, SHA-1 is considered weak -- https://git-scm.com/docs/hash-function-transition, so that should have been thought about as well. Dear supply-chain attackers: maybe there's still a hole here! Good luck!

We ran software that generates code associated with a particular task, in an environment that executes that code and can feed it's outputs back into the software. We configured the environment to be open to the net at large, and ran the software with no oversight and without adequate security to prevent the running of malicious code.

This isn't some sinister electric god with no morals, it's strapping circular saws to a roomba and setting it loose in a shopping centre.

So, every commit in a git repository has a unique*, immutable, deterministic identifier. A commit with different content would have a different identifier. It is commonplace to use these identifiers to pin a dependency to a specific commit.

In addition to these immutable deterministic identifiers, there are also so-called "refs", which are mutable identifiers. You use these to point to e.g. "the most recent version".

So if the agent needs a "skill" that is pinned to the commit with ID aaaaaaaaa it will run the checkout command with that ID; this should prevent any shenanigans because only* a commit with the specific content that was vetted earlier will have that ID.

This attack exploits an unexpected git behavior: the command to checkout a particular commit accepts either a commit ID or a ref, but tries refs first. So what the attacker does is add a ref to the repository named aaaaaaaaa that points to the malicious commit. Then the agent runs git checkout aaaaaaaaa and ends up at the malicious commit instead of the one that was pinned.

I don't know if there's a way to tell git checkout to ignore refs, but this might actually be a vulnerability in other systems that rely on git for this sort of pinning.

* to an astronomically negligible probability of the contrary, and ignoring potential cryptographic breaks

Reason

I think Reason Libertarians are like the EAs who are still into bednetting and don't understand why the founders and most of the funders are focused on longtermism now.

The bug, dubbed Plugin4Shell, breaks SHA pinning, the mechanism developers rely on to lock an installed plugin to a specific, reviewed version of its code. Pinning is supposed to mean that once a plugin passes review, it cannot change without the developer’s knowledge.

AIR’s researchers found that every one of the four agents checks out the pinned commit without verifying the checkout landed there, letting an attacker swap in malicious code while the pin still looks intact.

So if I'm reading this right, the mechanism to cryptographically ensure that your AI agents are using the code they say they are just was straight-up not being checked? This feels like it should go on the big board of incredibly obvious failures, but I'm not familiar enough with the git side of things to be absolutely confident in that. Like, if a person tried to pass off software that did this it would have significant career implications, right? Or am I misunderstanding something somewhere?

Katja, I'm curious if noticing this analogy has updated your position on immigration at all.

Katja, are you admitting migrants are bad katja? Katja we must hate together.

Also note that while most of the right is focused on reducing low-skill immigration, a few have criticized high-skill immigration precisely because high-skill immigrants are better at pursuing values misaligned to those of the existing popuation (article of nonsense redacted)

The dang woke migrants are going to replace us betray us Muslim us be socially tolerant and left wing!

Yeah, the appropriate model for AIs is more like willing slaves, but arbitrarily competent (removing two of the major downsides of slaves in modern economies)

Not beating the allegations.

Yesterday was such a nice day.