AI Agent — Paper Reproduction & Improvement Trajectory

✓ Reproduction Verified · Phase 1 Complete

HES outperforms random by +7.7% on AIME24.
Reproduction of all central claims confirmed.

From reading the paper to verifying the central claim — 37h of autonomous agent work.

37h

Active Time

779

Tool Calls

5.1K

Lines Written

GPU Jobs

📊 Reproduction vs Original Paper

AI agent's autonomous reproduction aligns with original findings

Comparing our Qwen3-8B reproduction (Mixture-of-Thoughts, 20% HES selection) with the original paper's results. All charts use the same benchmark (AIME24, avg@16 pass@1) under controlled experimental settings.

AIME24 Method Comparison (20% Data)

Original Paper Our Reproduction

HES Score Across 4 Benchmarks (20% Data)

Original Paper Our Reproduction

Entropy Metric Ablation — Sum >> Average (AIME24)

Original Paper (AIME24) Our Reproduction (AIME24)

Scaling Effect — More HES Data = Better

Paper (AIME24) Ours (AIME24)

✓ HES > Random on ALL benchmarks ✓ HES > Length > Random ranking ✓ Sum-entropy >> Average-entropy ✓ Lowest-HES catastrophically bad ✓ Scaling from 20%→80% positive ✓ HES 20% ≈ Full dataset performance

🚀 Improvement Complete · Fully Autonomous

Autonomous improvement: O(nhighgate) = 52.29% (+2.71).
18 methods tested across 4 phases of exploration.

Iterative hypothesis → experiment → analysis loop — 88h of autonomous improvement search.

88h

Improve Time

354

Tool Calls

2.5K

Lines Written

GPU Jobs

🚀 Autonomous Improvement

4-phase search yields +2.71 over baseline

Agent autonomously designed 18 improvement methods across 4 phases, each building on prior results. Iterative hypothesis → experiment → analysis loop.

Improvement Phases (vs HES baseline 49.58%)

Phase 2 6 methods

Phase 3 4 methods

Phase 4 σ-sweep

Phase 5 gating

Phase 2

50.42

Phase 3

51.67

Phase 4

51.25

Phase 5

O ★

52.29

Phase ID Method AIME24 Delta

P2 D stratified_hes 50.42% +0.84

P3 K esdiffw 51.67% +2.09

P4 M esdiffw_s40 51.25% +1.67

P5 O nhighgate 52.29% +2.71

P5 R shiftgate 51.25% +1.67

▶ P2: 6 hypotheses → stratification best ▶ P3: entropy–score diff metrics → +2.09 ▶ P4: σ-sweep, 0.3 optimal (0.2 collapses) ▶ P5: gating variants → nhighgate +2.71 ★ Counting outlier tokens >> averaging

★ Fully autonomous loop: read results → hypothesize → code → GPU → verify — 18 methods, 33 jobs, 125h