DeepSeek Harness vs Pi: What Bug Hunt Bench Results Show

A closer look at DeepSeek Harness, Pi and Claude Code on the same bug-fixing benchmark.

By Daniel Carter (pen name) · Reviewed November 9, 2026

An October 2026 benchmark compared DeepSeek V4.1 Flash 1017 across coding harnesses. DeepSeek Harness averaged 42.7 fixes out of 105 at MAX, Pi averaged 42.2, and Claude Code averaged 39.1. The reported costs and rankings describe that specific test, not every workflow.

Figures are historical test results for the same model and official API; they are not measurements of dssub. The scores are average bugs fixed out of 105, not percentages.

Two harnesses can send the same model at the same bug and produce different results. Bug Hunt Bench measures fixes across 105 bugs using a coding harness and the official API in the reported test. The results below describe an October 2026 snapshot.

What Bug Hunt Bench measures

Pawel Huryn’s Bug Hunt Bench tests harnesses on 105 bugs across two repositories. This comparison uses DeepSeek V4.1 Flash 1017 and reports results at LOW, HIGH and MAX effort. Settings and per-run costs belong to the tested configurations and are not dssub prices.

The comparison tested particular versions and configurations. Claude Code had its subagent tool switched off, while DeepSeek Harness used its shipped setting. Different configuration choices can affect the results.

Bug Hunt Bench October 2026 scores by effort tier: DSH leads MAX at 42.7 and LOW at 34.8, Pi leads HIGH at 40.4, and Claude Code trails in MAX and HIGH
DSH takes MAX and LOW. Pi takes HIGH. Claude Code does not lead a tier.

DSH leads two tiers in this comparison

The MAX result is the one DSH wants: 42.7 against 42.2 for Pi and 39.1 for Claude Code. DSH also takes LOW at 34.8, with Claude Code second at 34.4 and Pi third at 33.8. Pi takes HIGH at 40.4, which puts it 1.2 bugs on average ahead of DSH and 4 bugs on average ahead of Claude Code.

TierDeepSeek HarnessPiClaude CodeLeader
LOW34.833.834.4DSH
HIGH39.240.436.4Pi
MAX42.742.239.1DSH

Claude Code finishes third in MAX and HIGH and second in LOW. It trails the leader in every tier.

The ranking is tighter than the scores suggest

The DSH lead over Pi in MAX is 0.5 bugs fixed on average, while Pi leads DSH by 1.2 at HIGH. The run ranges overlap, so these small differences do not by themselves establish a reliable winner.

The top scores are close relative to the reported rerun variation. That does not establish a statistical tie; repeat the comparison on your own tasks before switching.

Planning to use DeepSeek in your workflow? Explore DeepSeek Subscription to review plans and usage limits.

Pi's advantage is the bill

Cost is where the tiers separate. In MAX, Pi spent $0.83 per run and DSH spent $1.84, so DSH cost about 2.2 times as much for 0.5 more bugs on average. In HIGH, Pi spent $0.64, which the benchmark author calls the best value in the set.

The published figures are cost per run, not cost per solved bug, so treat them as a budget signal rather than a unit price. The useful part is the direction: Pi reaches a comparable score with a much smaller bill, and that difference grows over a full day of runs.

MAX tier comparison: Pi costs 0.83 dollars per run and scores 42.2, DSH costs 1.84 dollars and scores 42.7, so Pi uses 45 percent of the budget for 98.8 percent of the score
Pi uses 45% of DSH's budget in MAX and keeps 98.8% of its score.

What this says about DSH and Pi

DSH is DeepSeek's own harness, and the MAX result fits a harness tuned around DeepSeek models. A team that builds around one model family can spend its tuning budget there. A general harness has to spread that budget across providers.

Pi costs less in this comparison and leads the other two selected harnesses at HIGH. Other harnesses also appear in the full benchmark: Grok Build CLI matched Pi at HIGH. Test the configuration that fits your own workflow.

How to use the result

  • Pick the tier you actually run. A MAX-tier win says little if you keep the harness on LOW for autocomplete.
  • Treat small gaps cautiously. Run-to-run variation can change the ordering.
  • Compare a week of runs, not a single run. The cost gap compounds with use.
  • Try one bug from your own repository before switching. Benchmark bugs are not your bugs.

FAQ

Did DeepSeek Harness win Bug Hunt Bench?

Among the three harnesses compared here, DSH led MAX and LOW, while Pi led HIGH. The complete benchmark included other harnesses; Grok Build CLI matched Pi at HIGH.

Is Pi better than DSH for coding?

It depends on the tier and budget. In this historical test, Pi was within 0.5 fixes of DSH at MAX for less than half the per-run cost. The run ranges overlap, so a small difference is not proof of superiority.

Does a higher harness score mean the model is better?

No. The score covers the whole harness configuration, including its model, prompts and tool loop. A harness tuned for one model can beat a general harness without the underlying model being stronger.

← Back to news