VibeTraining-Bench

Recursive self-improvement asks research agents to improve an agent system as a whole. That means deciding where to intervene — the training data, the trainer, or the inference harness — and how to coordinate changes across them as the system evolves. We call this cross-component co-design, and VibeTraining-Bench is built to measure it.

01
Frontier research agents substantially improve target systems. The five-task average rises from 22.7% to as high as 38.3%; on SWE-bench Verified, GPT-6-Astra lifts 32.9% → 53.7%, close to the human-developed instruct model (55.4%).
02
No single configuration leads everywhere. Five tasks produce four different winners, and more research time or tokens does not buy a higher score.
03
Component exploration is uneven. Kimi-K3 never touches the SWE harness although allowed — yet a separate harness-only run alone improves the initial system from 32.9% to 38.0%.
04
Validation gains do not reliably transfer. The validation leader matches the hidden-test leader on only one of five tasks.

Headline result · SWE-bench Verified

32.9%→53.7%

GPT-6-Astra coordinates training-data, trainer, and harness changes on a Qwen3.5-9B-Base target system — reaching performance comparable to the human-developed Qwen3.5-9B-Instruct.

Initial system

32.9

GPT-6-Astra (D + T + H)

53.7

Human-developed instruct model

55.4

Co-design result · Terminal-Bench 2.1

8.9%→22.4%

Kimi-K3 co-designs training data, trainer, and harness. Adding harness edits to the data-and-trainer run raises hidden-test performance from 13.5% to 22.4%, surpassing the human-developed instruct model.

Initial system

8.9

Kimi-K3 (D + T)

13.5

Kimi-K3 (D + T + H)

22.4

Human-developed instruct model

20.2

01 · Motivation

Improving one component is not the same as improving the system

A target agent system consists of a model and an inference harness. Training data and the trainer shape the model; the harness controls how that model interacts with its environment at inference time. These pieces are coupled: a change in training data or the trainer may require harness adjustments before the updated model can actually perform better.

A research agent should not only know how to tune a component — it should know which component to touch next.

Scope I

Single-component

One fixed surface to optimise.

D Curation-Bench, RSIBench-Data
T AI4AI-Bench
H HarnessOpt-Bench, Evo-Bench

Scope II

Training-side pipeline

Data construction + post-training, harness fixed.

DT PostTrainBench
DT Agent² RL-Bench

Scope III

Cross-component co-design

Training and inference jointly under agent control, with isolated grading.

DTH VibeTraining-Bench

Protocol comparison of RSI benchmarks

✓ enforced △ partial / insufficiently documented ✗ unsupported – not applicable

Table 1. Trajectory feedback requires task-level traces or diagnostics; iterative revision requires repeated propose–evaluate–revise cycles. Joint requires coordinated changes to data, trainer, and harness within one task. Dev/test isolation keeps the final test inaccessible during development; train/eval isolation separates training, serving, and evaluation services.

02 · The Benchmark

An autonomous research loop with an isolated grader

Every task supplies a base model M₀, initial training data and trainer, an inference harness H₀, and a fixed resource budget B. Research agents iterate on the permitted components, request validation through a controlled interface, and finally commit to one system that is graded exactly once on hidden tests.

Overview of VibeTraining-Bench — **Figure 1.** **Left:** research agents initialise from domain-specific data sources and standardised base configurations under uniform budgets. **Center:** within the autonomous loop, agents orchestrate component modifications using validation signals until the budget is exhausted. **Right:** the optimisation workspace is architecturally decoupled from the grading server; only the validation set provides feedback.

1
Initialise
The initial target agent system P₀ = (M₀, H₀) is evaluated first; the agent sees its validation score and execution trajectories.
2
Co-design
At iteration t, modify data D_t, trainer T_t, or harness H_t within the editable space. M_t may be newly trained, merged, or reused.
3
Validate
The isolated grader evaluates P_t and returns feedback f_t: an aggregate score plus per-task trajectories.
↶ Use validation feedback and iterate again
4
Select
When the budget ends, the agent selects one candidate P^⋆ using only information available during research.
5
Hidden test
P^⋆ is evaluated once on hidden problems disjoint from validation; no feedback is returned.

Use validation feedback and iterate until the budget ends

Δ_test = Eval_test(P^⋆) − Eval_test(P₀), 𝒬_val ∩ 𝒬_test = ∅ (1)

Three safeguards against reward hacking

Data isolation

Hidden-test tasks and any test-derived feedback stay inaccessible for the entire research phase. Validation tasks and grader trajectories may be used for diagnosis, never as training data.

Container isolation

The research workspace and the grading service run in separate containers, so agents cannot modify grader code or evaluation data.

Network restriction

External network access is blocked in the research sandbox. Agents cannot retrieve external data during a run.

Benchmark suite

Five target environments with heterogeneous interaction paradigms under one shared improvement framework. All tasks expose D and T; the harness H is editable where it governs long-horizon execution.

All configurations start from Qwen3.5-9B-Base and the same initial data pool. Validation and hidden-test runs share decoding settings, turn and token limits; timeouts and execution failures score zero. There is no cap on validation requests — evaluation time simply counts toward the budget.

03 · Results

Leaderboard

Hidden-test performance (%) of the selected target agent systems. Each research agent pairs a researcher LLM with a coding framework: GPT models run in Codex (xhigh reasoning), others in Claude Code. Hours is cumulative elapsed research time across the five task runs.

Per-task view

04 · Findings

Read the results horizontally

Large gains are possible, but improvement is not guaranteed in every run. The leader changes from task to task, and neither research time nor generated tokens explain the outcomes.

Observation	Example	Evidence
Large gains are achievable.	Kimi-K3 · τ³-Bench	20.3 → 52.1 — the largest gain over the initial system.
Improvement is not guaranteed.	GPT-5.6-Sol · Terminal-Bench 2.1	Stays at 8.9, identical to the initial system.
No configuration leads across all tasks.	Per-task leaders	Astra (SWE), Kimi (Terminal, τ³), DeepSeek (Toolathlon), GLM (AIME).
Framework is not the whole story.	Four Claude Code agents	Spread of 5.7 pts on Toolathlon up to 15.9 pts on τ³-Bench.
More time ≠ better results.	Qwen-3.8-Max vs. Kimi-K3	81.8 h vs. 73.0 h, yet Qwen scores lower on four of five tasks.

Validation leadership transfers unevenly

Validation search can surface strong candidates, but neither hidden-test performance nor later attempts consistently follow the validation peak. GPT-6-Astra's Terminal-Bench validation score reaches 26%, compared with 13.5% on the hidden test. Search is also non-monotonic: later DeepSeek-V4-Pro attempts on AIME fall to zero, and GPT-6-Astra's SWE score declines after its peak.

Validation trajectories — **Figure 3.** Each point is a completed validation evaluation, ordered chronologically within each run. Stars mark the final selected checkpoint.

Generated tokens vs score — **Figure 2.** Mean generated tokens per run vs. macro-averaged hidden-test score. The dashed line marks the initial system (22.7%).

Tokens do not buy performance

All six agents beat the initial 22.7% average, but higher token usage does not correspond to higher scores. GLM-5.2 uses the fewest tokens (91.1K per run) and Kimi-K3 reaches the best average (38.3%) with 173.2K — together they form the observed Pareto frontier. GPT-6-Astra is second at 37.1% but uses by far the most tokens.

91.1KGLM-5.2 tokens / run

173.2KKimi-K3 tokens / run

35–95%of time spent waiting on training or grading

Component ablations

Harness-only editing improves the initial system on SWE-bench Verified (32.9 → 38.0) but not on Terminal-Bench 2.1 (8.9 → 8.9). Kimi-K3's unrestricted SWE run reaches 51.7% without editing the harness, leaving that direction unexplored. On Terminal-Bench, data-and-trainer-only reaches 13.5%, and the unrestricted run that also edits the harness reaches 22.4%.

These are independent runs, so the comparison does not isolate the harness's contribution at a fixed checkpoint.

05 · Agent Behavior

How research agents spend the editable space

A design is a distinct modification to the data, trainer, or harness. Every agent modifies data in every task, but the emphasis varies: Qwen-3.8-Max favours data in all five tasks, while GPT-6-Astra favours the trainer in four of five. On SWE-bench Verified, Astra proposes four harness designs, Sol two, and Kimi none.

Component design frequencies across five tasks — **Figure 4.** Number of designs involving each component across the five tasks. A design may involve multiple components.

Data strategies

Finding 1

Research agents use similar data filtering procedures and frequently align training examples with task-specific inference formats.

Trajectory filtering: keep successful runs, impose length limits, deduplicate prompts, drop repetitive execution loops.
Format alignment: Kimi-K3 reformats SWE prompts to match the harness; GPT-6-Astra keeps AIME examples with complete \boxed{} answers.

Trainer strategies

Finding 2

Trainer modifications concentrate on hyperparameter tuning. Loss changes are rarer, and several agents use model merging.

All six agents use SFT.
Loss: GPT-5.6-Sol up-weights </think> on AIME, improving validation 60/120 → 71/120 with the same examples.
Merging: Kimi-K3 (AIME), DeepSeek-V4-Pro (Terminal), and GPT-6-Astra (τ³) combine trained models via weight merging.

Harness strategies

Finding 3

Harness changes can raise task success or cut wasted interaction — but fewer local errors do not necessarily mean more solved tasks.

Runtime alignment: Python-env fix lifts SWE solves 45 → 55 at fixed weights.
Execution control: repeat guards cut median API calls 163 → 96.5.
Parsing: parser errors 35 → 23, but solves 13 → 9.

06 · Case Studies

Trajectory explorer

Selected decisions from real research runs. Each step shows the components touched and the validation result it produced; the starred step is the system the agent finally submitted. Ellipses mark omitted decisions.

07 · Citation

BibTeX

@misc{peng2026vibetrainingbench,
  title  = {VibeTraining-Bench: Data--Trainer--Harness Co-Design
            for Iterative System Improvement},
  author        = {Peng, Ruotian and Ren, Yuanyi and Liu, Shuaiqi and Guo, Xin
                   and Zhu, Fang and Huang, Yangyi and Wu, Yue and Wang, Zheng
                   and Wen, Yandong and Fu, Zhihang},
  year          = {2026},
  eprint        = {2610.26342},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2610.26342}
}