“ecologically valid evals” is a phrase that will be very important. Up there with “inter-rater reliability”
One of the most important (and underdiscussed) facts about AI "swarms" is that they are extremely, extremely expensive.
— Tom Reed (@mentalgeorge) September 6, 2026
Consider that METR's *investigation* of the HF hack transcripts cost $400k in API credits. Presumably the actual swarm burned through a whole OOM (or two?) more
An important implication is that there'll a growing bifurcation in AI workloads between
1. the everyday coding and engineering tasks that most customers use AI for;
2. Large, expensive moonshot programmes that a smaller set of orgs can afford to turn a multimillion dollar swarm-cannon at.
I'd predict that the serious future "warning shots" are likely to come from (2).
I also suspect that aligning models for (1) and aligning models for (2) is really quite different. The distribution for (1) is better-understood, easier to model and build ecologically valid evals for, whereas anything in (2) is almost by definition OOD. It's also going to be far more costly aligning agents on tasks in (2), and the relevant time-horizons are more brutal for model release cycle.
The "good news" is that, for quite some time, the pool of actors capable of targeting swarms at anything in (2) will be much smaller. In theory, it is much more practicable for labs to know who's launching swarm-cannons - you could imagine some AI ID schemes only applying to ultra-large-inference projects, for instance, rather than just everyday use.