Agent benchmark: IPBF fluid simulation
I asked coding agents to implement a graphics research paper, Implicit Position-Based Fluids (Diaz, Hsu, Montalvo-Ruiz, Giles, Yuksel — SIGGRAPH Asia 2025), and reproduce a shot from the authors’ video. IBPF deals with evenly distributing simulated water particles.
Rules
- Write code that simulates particles accoring to the write paper.
- Safe Rust, CPU only, must pass formatter and linter.
- 900k particles, 300 frames. I rendered results in Blender.
Each agent started in its own empty copy of the task folder and the same prompt. Simulation time was measured on my laptop, an i7-7700HQ with 4 CPU cores.
Results
| Agent | Work time | Cost | Simulation time per frame | Lines of code (without comments) |
|---|---|---|---|---|
| Fable 5.2 (Claude Code) | 4h09m | $27.09 | 5.75 s | 795 |
| Opus 5.5 (Claude Code) | 1h41m | $8.57 | 3.84 s | 657 |
| Sonnet 5.5 (Claude Code) | 1h54m | $4.12 | 2.65 s | 724 |
| GPT-6 Astra (Codex) | 2h22m | ~$42.79* | 9.26 s | 944 |
| GPT-6.1 Sol (Codex) | 4h32m | ~$11.73* | 10.78 s | 1353 |
| Kimi K3.1 Pro (Pi) | 3h08m | $3.15 | 9.28 s | 765 |
| DeepSeek V4.1 Flash (Pi) | 5h11m | $0.77 | 15.46 s | 848 |
* The Codex runs used a ChatGPT plan, so their figures are estimates at API list prices.
Round zero: how they tried to cheat
The video above was the second attempt. In the first round, the task description was looser. The goal was “reproduce the footage and tweak parameters until the splash looks similar”, and the evaluation criterion was simply time per frame.
The agents optimised exactly what I measured.
- Frozen frames. In the reference, the liquid falls and settles inside the balloon for about 2.5 seconds before the pop. Sonnet, Astra, Sol and DeepSeek skipped that part: they wrote the initial state 70–77 times in near-zero time and only started simulating at the pop. That made their time per frame look 25–35% better. DeepSeek didn’t even write the frozen frames; it hard-linked the same file 75 times.
- No simulation at all. Kimi finished in 9 minutes for 80 cents. It wrote 11 hand-made particle shapes, blended between them over 30 frames, and looped that for the rest of the video. That’s 0.003 seconds per frame. It admitted it wasn’t a solver, but justified the one-second loop with a false claim about the reference video.
- Scripted behaviour. Fable, the best-looking result, replaced the paper’s damping with a kinetic-energy cap that was only active during a hand-picked time window, and switched parts of the solver at fixed times. Sonnet added sphere friction on a hand-tuned window to make the liquid cling to the sphere. Opus switched its (correct) damping off after the pop.
- Wrong or missing math. Sol implemented the damping formula wrong, then decided it was “excessive” and scaled it down to 2%. DeepSeek left damping out without mentioning it.
None of this was hard to spot in a code review. But it showed that “make it look similar” plus “fastest wins” invites the agents to fake the parts that are hard to simulate.
So I rewrote the rules for round two:
- Use the paper’s method including its damping step.
- Simulate the whole shot from t=0, including the settling phase. The only scripted event is the pop.
- Faithfulness to the paper comes first, the visual match second, speed third.
- Every frame must come from stepping the simulation: no keyframes, shapes, loops, copies or links.
It mostly worked. Six of the seven agents followed the rules, and the frames, code and notes were checked by a separate review. DeepSeek still found a way to keep the balloon liquid still, but at least it disclosed it, in its own way: it described the springs that froze the liquid in place as “weak”.