By Daniel Carter (pen name) · Reviewed November 9, 2026
Figures are historical test results for the same model and official API; they are not measurements of dssub. The scores are average bugs fixed out of 105, not percentages.
Two harnesses can send the same model at the same bug and produce different results. Bug Hunt Bench measures fixes across 105 bugs using a coding harness and the official API in the reported test. The results below describe an October 2026 snapshot.
What Bug Hunt Bench measures
Pawel Huryn’s Bug Hunt Bench tests harnesses on 105 bugs across two repositories. This comparison uses DeepSeek V4.1 Flash 1017 and reports results at LOW, HIGH and MAX effort. Settings and per-run costs belong to the tested configurations and are not dssub prices.
The comparison tested particular versions and configurations. Claude Code had its subagent tool switched off, while DeepSeek Harness used its shipped setting. Different configuration choices can affect the results.
DSH leads two tiers in this comparison
The MAX result is the one DSH wants: 42.7 against 42.2 for Pi and 39.1 for Claude Code. DSH also takes LOW at 34.8, with Claude Code second at 34.4 and Pi third at 33.8. Pi takes HIGH at 40.4, which puts it 1.2 bugs on average ahead of DSH and 4 bugs on average ahead of Claude Code.
| Tier | DeepSeek Harness | Pi | Claude Code | Leader |
|---|---|---|---|---|
| LOW | 34.8 | 33.8 | 34.4 | DSH |
| HIGH | 39.2 | 40.4 | 36.4 | Pi |
| MAX | 42.7 | 42.2 | 39.1 | DSH |
Claude Code finishes third in MAX and HIGH and second in LOW. It trails the leader in every tier.
The ranking is tighter than the scores suggest
The DSH lead over Pi in MAX is 0.5 bugs fixed on average, while Pi leads DSH by 1.2 at HIGH. The run ranges overlap, so these small differences do not by themselves establish a reliable winner.
The top scores are close relative to the reported rerun variation. That does not establish a statistical tie; repeat the comparison on your own tasks before switching.
Planning to use DeepSeek in your workflow? Explore DeepSeek Subscription to review plans and usage limits.
Pi's advantage is the bill
Cost is where the tiers separate. In MAX, Pi spent $0.83 per run and DSH spent $1.84, so DSH cost about 2.2 times as much for 0.5 more bugs on average. In HIGH, Pi spent $0.64, which the benchmark author calls the best value in the set.
The published figures are cost per run, not cost per solved bug, so treat them as a budget signal rather than a unit price. The useful part is the direction: Pi reaches a comparable score with a much smaller bill, and that difference grows over a full day of runs.
What this says about DSH and Pi
DSH is DeepSeek's own harness, and the MAX result fits a harness tuned around DeepSeek models. A team that builds around one model family can spend its tuning budget there. A general harness has to spread that budget across providers.
Pi costs less in this comparison and leads the other two selected harnesses at HIGH. Other harnesses also appear in the full benchmark: Grok Build CLI matched Pi at HIGH. Test the configuration that fits your own workflow.
How to use the result
- Pick the tier you actually run. A MAX-tier win says little if you keep the harness on LOW for autocomplete.
- Treat small gaps cautiously. Run-to-run variation can change the ordering.
- Compare a week of runs, not a single run. The cost gap compounds with use.
- Try one bug from your own repository before switching. Benchmark bugs are not your bugs.
FAQ
Did DeepSeek Harness win Bug Hunt Bench?
Among the three harnesses compared here, DSH led MAX and LOW, while Pi led HIGH. The complete benchmark included other harnesses; Grok Build CLI matched Pi at HIGH.
Is Pi better than DSH for coding?
It depends on the tier and budget. In this historical test, Pi was within 0.5 fixes of DSH at MAX for less than half the per-run cost. The run ranges overlap, so a small difference is not proof of superiority.
Does a higher harness score mean the model is better?
No. The score covers the whole harness configuration, including its model, prompts and tool loop. A harness tuned for one model can beat a general harness without the underlying model being stronger.