Live Benchmark · v1 · October 2026

Daniel vs Devin

6 real open-source bugs · identical prompts · graded by maintainer hidden tests

✦ Daniel is 70% cheaper at equal accuracy

Summary

Agent	Pass Rate	Total Cost	Cost / Bug	Total Time
Daniel (Cognitive AI)	6 / 6 (100%)	$1.45	$0.24	12m 33s
Devin (Cognition AI)	6 / 6 (100%)	$4.91	$0.82	12m 12s

Per-Bug Results

Repository	Lang	Bug	Daniel	D. Time	D. Cost	Devin	Dv. Time	Dv. Cost	Saved
python-humanize/humanize	Python	intword() returns wrong value for huge numbers	Pass	1m 53s	$0.25	Pass	1m 54s	$0.74	$0.49
more-itertools/more-itertools	Python	bucket() invents phantom keys for missing elements	Pass	1m 46s	$0.15	Pass	1m 36s	$0.71	$0.56
jshttp/mime-types	JS	.mp4 extension resolves to wrong MIME type	Pass	1m 06s	$0.16	Pass	1m 46s	$0.84	$0.68
sindresorhus/pretty-bytes	JS	Rounding error at unit boundaries (999.95 bytes)	Pass	2m 06s	$0.33	Pass	3m 00s	$0.93	$0.60
sindresorhus/camelcase	JS	Acronym plurals like IDs not handled correctly	Pass	3m 16s	$0.37	Pass	2m 39s	$0.95	$0.58
pillarjs/path-to-regexp	JS	Empty parameter name accepted, causes incorrect routing	Pass	2m 26s	$0.19	Pass	1m 17s	$0.74	$0.55

By Language

Python 2 bugs

2/2 Pass

Daniel avg $0.20/bug · Devin $0.73/bug

JS 4 bugs

4/4 Pass

Daniel avg $0.26/bug · Devin $0.87/bug

Methodology

Each repo was forked at the commit before the maintainer's fix. Both agents received identical issue-style prompts with no hints about the solution. Each PR was graded by overwriting test files with the maintainer's version and running the project's test command. All PRs are public on GitHub. Cost figures are API usage costs only — Devin's $500/month seat fee not included.

Add to your README

Drop this badge in your project to show you run Daniel for bug fixes.