From Fleet to Flag: An LLM Agent that Autonomously Solves a CTF
I spent a week trying to get a machine to solve a Hack The Box web challenge on its own — no human feeding it the vulnerability, no peeking at the writeup, just source code, a live target, and a model (sometimes handed a lead by another model, never by me). This post is the honest account of what worked, what didn't, and what the numbers actually say about model reasoning, hypothesis seeding, and the role of the harness around the model.
This is not a challenge writeup
The CTF is only the measuring stick. What I actually care about here is the models and the harness: how architecture around a model decides whether it can reason at all, where stronger models pull ahead, how one model's hypothesis accelerates another, and what it costs. The vulnerability details appear only as evidence for those claims — if you want the clean exploit, the payload is in one code block and you can leave.
The headline: a mid-tier open model (DeepSeek v4-pro), handed another model's hypothesis and a
thin single-agent loop, autonomously converted it into a working Mongoose prototype-pollution
exploit and pulled a live flag for $1.81 — the exact payload every other configuration circled
but missed.