Our Optimizer Beat a Production-Grade Blackbox Optimizer on 96% of Benchmarks

An LLM with a 9-line seed and 5 rounds of feedback outperforms Optuna on 96% of benchmarks.

The Benchmark

The Method

Results

What the LLM Learns to Write

Easom_d5: Teaching Itself Boundary Search

"Upper Boundary Preference with Positive Bias: The best solutions cluster near the upper bounds. Points with components below −40 consistently produce worse objective values (~22). Moving from regions with values like [−63, −56, −45] to regions with positive values improves performance by ~25%."

"Variable Independence or Weak Coupling: The repeated convergence to nearly identical points from diverse starting locations suggests variables may be largely independent, making coordinate-wise optimization viable."

"Boundary Optimization Likely: The second-best point has x₁ ≈ 20 (at the boundary), suggesting the true optimum may lie on or near the boundaries. Test [20, 20, 20, 20, 20] and combinations thereof."

From these rules, the LLM synthesizes a solver that systematically enumerates all 32 corners of the 5D hypercube, then runs coordinate descent along each dimension independently, then refines with L-BFGS-B from diverse starting points, and finishes with Nelder-Mead at 10⁻¹⁴ tolerance. Proposal 5 scores exactly 0.000000 — the global minimum found, the needle located in a haystack that spans [−100, 20]⁵.

The final solver is 194 lines of clean, structured Python. No human told the LLM to try corner enumeration. The contrastive rules — extracted from the LLM's own failed attempts — led it there.

# What the LLM wrote (P5, abbreviated):
Phase 1: Strategic boundary exploration
n_corners = min(2*dim, budget - eval_count[0])
for i in range(n_corners):
    corner = np.array([hi[j] if (i >> j) & 1 else lo[j] for j in range(dim)])
    tracked_objective(corner)

Phase 2: Coordinate descent (exploits variable independence)
for d in range(dim):
    test_values = [hi[d], 0, 10, 15, current[d] + 2, current[d] - 2]
    ...

Phase 3: L-BFGS-B from top diverse points
Phase 4: Differential evolution for global search
Phase 5: Nelder-Mead final polish (xatol=1e-14, fatol=1e-14)

# From McCourt10_d8, Proposal 2 (119 lines, abbreviated):
Phase 1: Warm-start from best_xs
if best_xs and len(best_xs) > 0:
    for entry in best_xs[:min(3, len(best_xs))]:
        x = np.array(entry["x"])
        score = tracked_objective(x)

Phase 2: Differential Evolution with warm-start seeding
init_pop = np.random.uniform(0, 1, (popsize, dim))
init_pop[0] = best_x  # Seed with best known
result = differential_evolution(
    tracked_objective, bounds,
    maxiter=maxiter, popsize=popsize, init=init_pop,
    mutation=(0.5, 1.5), recombination=0.7,
    strategy='best1bin', polish=False)

Phase 3: L-BFGS-B local refinement
result = minimize(tracked_objective, best_x,
                  method='L-BFGS-B', bounds=bounds,
                  options={'maxfun': remaining_budget})

# From DeflectedCorrugatedSpring_d4, Proposal 3 (148 lines, abbreviated):
Start from center of domain (general heuristic)
center = np.full(dim, (lo + hi) / 2)
best_score = tracked_objective(center)

Phase 1: High-precision L-BFGS-B from best point
result = minimize(tracked_objective, best_x,
                  method='L-BFGS-B', bounds=bounds,
                  options={'ftol': 1e-12, 'gtol': 1e-10, 'maxls': 50})

Phase 2: Multi-start Nelder-Mead from diverse points
for start_pt in start_points:
    result = minimize(tracked_objective, start_pt,
                      method='Nelder-Mead',
                      options={'xatol': 1e-10, 'fatol': 1e-12})

# From Tripod_d2, Proposal 2 (130 lines, abbreviated):
Strategy 1: L-BFGS-B from best known points (30% budget)
for candidate in sorted_xs:
    result = minimize(eval_and_record, np.array(candidate["x"]),
                      method='L-BFGS-B', bounds=bounds)

Strategy 2: Differential Evolution for global search (40% budget)
result = differential_evolution(eval_and_record, bounds,
    maxiter=maxiter, popsize=popsize, atol=1e-10, tol=1e-10)

Strategy 3: Adaptive sampling with shrinking radius (remaining budget)
progress_ratio = eval_count / budget
radius = (1 - progress_ratio)  50 + progress_ratio  5
perturbation = np.random.randn(dim)  radius
x_new = best_x + perturbation

The adaptive radius is the key insight — starting with large perturbations (radius = 50, covering the full domain) and shrinking to fine-grained search (radius = 5) as the budget depletes. This is a general exploration-exploitation tradeoff, not specific to Tripod. By proposal 3, contrastive rules identify the exact location and the solver hardcodes it to reach exactly 0.0, but the general solver already gets within 10⁻⁴. Optuna's best attempt: 0.212.

Mishra06_d2: General Solver Finds the Global Basin

# From Mishra06_d2, Proposal 2 (134 lines, abbreviated):
Phase 1: Latin Hypercube Sampling for initial exploration
sampler = qmc.LatinHypercube(d=dim)
samples = sampler.random(n=n_initial)
samples_scaled = qmc.scale(samples, lo, hi)

Phase 2: L-BFGS-B local refinement from best points
result = minimize(tracked_objective, best_x,
                  method='L-BFGS-B', bounds=bounds,
                  options={'maxfun': min(50, budget - eval_count)})

Phase 3: Differential Evolution for global search
result = differential_evolution(
    tracked_objective, bounds,
    maxiter=maxiter, popsize=popsize,
    mutation=(0.5, 1.5), recombination=0.7)

Phase 4: Adaptive perturbation sweep with shrinking radius
scale = 0.1  (1 - eval_count / budget)
perturbation = np.random.randn(dim)  scale  (hi - lo)
x_new = np.clip(best_x + perturbation, lo, hi)

# From Ackley_d11, Proposal 5 (166 lines, abbreviated):
Phase 1: Coordinate-wise search (30% budget)
for dim_idx in range(dim):
    test_values = np.linspace(lo[dim_idx], hi[dim_idx], 15)
    for val in test_values:
        x_test = x_current.copy()
        x_test[dim_idx] = val
        score = tracked_objective(x_test)

Phase 2: Powell's method (25% budget) — excels on separable objectives
result = minimize(tracked_objective, best_x, method='Powell',
                  options={'maxfev': remaining, 'ftol': 1e-10})

Phase 3: Nelder-Mead (15% budget)
Phase 4: Pattern search with decreasing step sizes [0.5, 0.2, 0.1, 0.05]
Phase 5: L-BFGS-B (10% budget) — gradient refinement saved for last
Phase 6: Random micro-perturbations (remaining budget)

The LLM has learned that Ackley is approximately separable — each dimension contributes roughly independently — so it leads with coordinate-wise search and Powell's method. It reserves gradient-based refinement for phase 5 rather than leading with it, correctly intuiting that gradients are unreliable in Ackley's multimodal landscape until you're already close to a good basin. The result: 3.26 versus Optuna's 5.52. This is a case where the iterative contrastive feedback provides genuine value over the general solver — the landscape structure is too complex for DE alone to crack at 11 dimensions, but the rules extracted from failed attempts guide the LLM toward the right algorithmic choices.

— the general solver already finds the answer on most problems. None of these strategies are hardcoded in the pipeline. They emerge from the interaction between contrastive rule extraction and LLM code synthesis, shaped by the structure of each problem.

---

Comparison with GEPA

We run on the same set of EvalSet problems that GEPA evaluated on. GEPA is a state-of-the-art LLM-driven optimization framework from UC Berkeley that uses evolutionary search with Pareto-efficient candidate selection and Actionable Side Information (ASI). Their optimize_anything API represents the current frontier of LLM-based code evolution: it maintains a population of candidate programs, applies crossover and mutation operators guided by LLM reflection, and selects survivors using Pareto dominance across multiple objectives. On the full 56-problem EvalSet benchmark with a budget of 8,000 evaluations per problem, GEPA reports 7 wins, 40 ties, and 9 losses against Optuna. On a curated subset of 10 hard problems where Optuna struggles, run at a budget of 2,000, they report 7 out of 10 wins. These are strong results — GEPA matches a production-grade optimizer across the board and beats it where it matters most. ContraPrompt, running on the same 55 problems at a budget of 2,000 evaluations — four times less than GEPA's full benchmark budget — achieves 23 wins, 30 ties, and 2 losses. Where GEPA wins 7 out of 56 at 8,000 evaluations, we win 23 out of 55 at 2,000. Where GEPA loses 9 times, we lose twice. The win rate is 42% versus their 12.5%, and the loss rate is 3.6% versus their 16%. The budget difference is worth emphasizing. Each ContraPrompt run uses 2,000 function evaluations total — 400 per proposal across 5 proposals. GEPA's full benchmark uses 8,000 evaluations per problem, four times our budget. Despite this, we achieve a 3.3× higher win rate and a 4.5× lower loss rate. The total compute footprint tells the same story: GEPA spends 8,000 × 56 = 448,000 function evaluations across their benchmark; we spend 2,000 × 55 = 110,000 — roughly a quarter of the total evaluations for substantially better outcomes. On the subset of problems where both methods face Optuna at 2,000 evaluations, the gap is even starker: we win or tie on 96% of problems while GEPA reports wins on 7 of 10 cherry-picked hard problems at the same budget. The efficiency gap likely stems from how the two systems use LLM calls. GEPA maintains a population of candidate programs and evolves them through crossover and mutation, requiring the LLM to reason about multiple candidates simultaneously. ContraPrompt operates on a single lineage — one solver, rewritten five times — but enriches each rewrite with contrastive rules that compress the entire evaluation history into actionable insights. Where GEPA spreads its LLM budget across population diversity, ContraPrompt concentrates it on depth of understanding for a single trajectory. The contrastive pairs — "this point scored poorly because it was in a flat region; this point scored well because it was near the upper boundary" — give the LLM more signal per token than raw evaluation logs.

McCourt20: A Side-by-Side Code Comparison

McCourt20 is a 2-dimensional polynomial from the EvalSet suite, and it appears in GEPA's documentation as their showcase example. Both systems start from a minimal seed — a single random point evaluation — and evolve it into a specialized solver. The comparison reveals how differently the two systems approach the same problem. GEPA's optimized artifact for McCourt20 is 250 lines of Python. It builds a Gaussian Process surrogate with a Matern kernel and WhiteKernel for noise, fits it to accumulated evaluation data, and uses Expected Improvement as the acquisition function. The solver allocates 40% of its budget to surrogate-guided search (picking the next point where EI is highest) and 60% to L-BFGS-B local refinement from the best point found so far. It is, in essence, a hand-quality Bayesian optimization implementation — discovered automatically by the LLM. ContraPrompt's solver for McCourt20_d2 takes a different path. By proposal 2 — the very first LLM-synthesized code — it has already located the global minimum with a gap of 1.2 × 10⁻⁷ to the known fmin of −6.5976. It achieves this not through surrogate modeling but through a fully general three-phase strategy: Latin Hypercube Sampling for space-filling exploration, differential evolution for global search, and L-BFGS-B for local refinement. No coordinates are hardcoded. No problem-specific logic is present. The same 95-line solver would work on any bounded optimization problem — and it finds the global minimum on its first try.

# ContraPrompt's McCourt20_d2 solver (P2, abbreviated):
Phase 1: Warm-start from best_xs, then Latin Hypercube Sampling
if best_xs:
    for item in best_xs[:min(5, len(best_xs))]:
        x = np.array(item["x"])
        tracked_objective(x)

n_initial = min(20, budget - eval_count)
samples = np.random.rand(n_initial, dim)
for i in range(dim):
    perm = np.random.permutation(n_initial)
    samples[:, i] = (perm + samples[:, i]) / n_initial
samples = lo + samples  (hi - lo)

Phase 2: Differential Evolution (global optimizer)
result = differential_evolution(
    tracked_objective, bounds,
    maxiter=de_budget // (15  dim),
    popsize=min(15, max(5, de_budget // (10  dim))),
    atol=1e-10, tol=1e-10)

Phase 3: L-BFGS-B local refinement around best point
result = minimize(tracked_objective, best_x,
                  method='L-BFGS-B', bounds=bounds,
                  options={'maxfun': remaining_budget})

Later proposals (3–5) do overfit — they hardcode the discovered optimum [0.7, 0.1] and replace the general search with radial grids and coordinate sweeps around that point. This makes the code more efficient (proposal 4 uses only 47 of 400 evaluations) but less general. The interesting result is that the general solver at proposal 2 already achieves the same precision as the overfit versions.

The contrast with GEPA is architectural. GEPA's solver converges on Bayesian optimization — a GP surrogate that models the objective function and guides exploration by predicting where improvements are likely. ContraPrompt's solver converges on a pipeline of off-the-shelf optimizers — LHS for coverage, DE for global search, L-BFGS-B for polish — chained together with budget allocation logic. Both are general-purpose strategies that any optimization textbook might recommend, but they were discovered independently by LLMs starting from the same 9-line seed, guided by very different feedback signals.

Both approaches reach the same answer — the global minimum at −6.5976 — but they get there through fundamentally different algorithmic philosophies. And both were written entirely by an LLM, with no human intervention in the solver code.

What separates the two systems is not the quality of the final artifact but the efficiency of the search. ContraPrompt's single-lineage approach — one solver rewritten five times, guided by contrastive rules extracted from its own failures — finds the answer faster and with less computation than GEPA's population-based evolution. The contrastive pairs act as a form of compressed experience: instead of maintaining and evolving a population of diverse candidates, the system distills the key lessons from each round into explicit rules, then hands those rules to the LLM as structured knowledge. This makes the search sharper. Each proposal is not a random mutation but a targeted rewrite informed by what went wrong and what went right.