Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?

Joint work with Kidus Seyoum. The paper is on arXiv.

There’s been a lot of talk lately about recursive self-improvement: AI systems that improve the systems they run on, with each round making the next one easier. In my last post I pointed to self-evolving harnesses as one place where this is already happening, with agents rewriting their own scaffolding for measurable gains. This project started as a way to see how that would play out in hardware. Hardware design verification (DV) is one of those domains models see far less of in training than software engineering, and the results are easy to check, so it’s a good test of whether the loop keeps improving or stalls.

Setup

We keep the subject model fixed and have stronger language models like Opus 5 and GPT-5.5 propose changes to the runtime harness and scaffold configuration, based on execution evidence. Candidates are screened with cheap trials, then confirmed with five-trial runs. Every candidate is materialized in its own directory, evaluated, and recorded in an archive.

The main benchmark is 12 proprietary DV root-cause localization tasks. The agent gets a read-only workspace and has to find where the bug is and recommend a change. The primary subject model is Kimi K3, with additional runs on DeepSeek V4 Pro and MiniMax M3.

What we found

Evolution helps a lot with finishing and a little with being right. The evolved harnesses raised completed attempts by 71–76% (34/60 → 58–60/60) and any-hit task coverage by 80–100% (5/12 → 9–10/12). Total correct attempts rose only 18–24% (17/60 → 20–21/60). The mechanisms the proposer came up with, like bounded search, evidence-grounded queries, coverage ledgers, and causal commitment, mostly keep the agent from stalling or running out of turns.

The gains don’t consolidate. The two best evolved harnesses solved 11/12 tasks between them, but neither one was strongest everywhere. One solved T3 but failed every T10 trial; the other recovered T10 and lost T3. A larger run produced 958 candidates, and the archive as a whole covered 10/12 tasks (the baseline passed 1/12), but only two candidates passed our strict promotion rule. We also saw validation gains fail to transfer: a DeepSeek harness went from 0.375 to 0.725 on a held-out validation set, then tied the baseline at 2/12 on full replays.

It works for repair too. In a case study on 142 CVDP tasks with GLM-5.2, an evolved repair harness reached 61/142 functional passes against a 45/142 baseline, 35.6% more. The final functional verifier ran only after the agent’s output was complete, and its result was not fed back to the agent for that task.

Takeaway

If evolution keeps producing specialists instead of one harness that wins everywhere, then picking a single “best” harness and throwing away the rest loses useful work. We argue the archive should be a first-class output: keep manifests, task-level scores, traces, and lineage; select or route among complementary harnesses; and report archive-union coverage separately from single-harness performance so it doesn’t look like one harness is better than it is.

This also matters for recursive self-improvement. The usual picture has each round building on the one before, so improvements stack up. In our setting, each round mostly found a different specialist instead. A self-improving loop may need a step that explicitly merges or routes between what earlier rounds found. Otherwise it keeps rediscovering partial wins instead of compounding them.

The usual caveats apply. Twelve tasks is a small benchmark, five trials per task still leaves noise, and each evolved harness bundles several changes, so we can’t attribute gains to any single edit. The paper covers these in more detail.

Paper: arXiv:2609.28908