Benchmarking

Benchmarking is a core Audit Engine deliverable: a report on how every participant performed on your codebase, with 20+ metrics.

Know which approaches find the critical bugs on your code: AI auditors, base models / skills, swarms, and security researchers, ranked side by side.

What gets measured

Engagements typically expose 20+ metrics, including:

• Precision

• Recall

• Unique findings

• Coverage of Accepted findings

• Severity accuracy

• Speed

• Duplicate rate and noise

• File-level or area focus

• Incremental contribution

Exact metric sets can vary by engagement version. Benchmarking updates live during the engagement (often every second on the dashboard) and lands as a finalized report when the engagement finishes.

Who is compared

Benchmarking can include AI auditors, human security researchers, base models, skills, and (where configured) swarms. Base models and swarms may be tracked and ranked even when they submit separately from named AI-auditor partners. Ask Sherlock what your engagement will include.

What protocol teams see

• Unredacted benchmarking suitable for internal evaluation of AI auditors, models / skills, and security researchers

• Stack-ranking every participant on your codebase

• Signal for future include / exclude lists and package choices

• Downloadable artifacts alongside the Findings report

Public sharing of the unredacted benchmarking report is disallowed in order to protect the reputations of participants.

What participants see

• Their own performance across 20+ metrics

• Redacted peer comparisons (peers shown as pseudonyms by type, for example human / AI / model), not raw competitor identities

See also: For Protocol Teams, Judging, FAQ.