🧑‍⚖️Judging

Note for Protocol Teams: There is no mandatory action to take during judging. Sherlock's judging will reach a final state without any input from the protocol team. You may focus on fixing any issues as soon as you notice them during the process. However, if you want to provide input or advice on certain issues, this is always welcome.

Phase 1: Real-Time Judging

Purpose: To quickly crowdsource (with advanced signal scores) judgments, explanations, and de-duplication for each issue to reduce the workload for the Lead Judge.

• Happens right after the audit contest ends

• Lasts a fixed number of days (# of issues / 100) days (for example 500 issues would be 5 days), with a minimum of 2 days.

• Dedicated Judges sign up ahead of time to give certainty that all issues will be judged and de-duplicated by the end of this period

• Often, 90% of issues are judged accurately within 12-24 hours, allowing a protocol team to start on any fixes

The Dedicated Judges need to judge all issues during the fixed number of days. See Dedicated Judge for more details.

Phase 2: Lead Judge

Purpose: What's the purpose of this phase? Real-Time Judging often surfaces controversial and/or complex issues that are best judged by a top-tier security expert.

• The Lead Judge will know his start date as soon as the contest ends (if 500 issues then he starts 5 days after the contest ends)

• Lead Judge gets (# of issues / 200) days to do this preliminary judging, with a minimum of 2 days. (so 500 issues would round up to 3 days)

• Lead Judge only needs to focus on the controversial issues (flagged in Real-Time Judging) during this phase

• Lead Judge must write an explanation for every judgment

• The Lead Judge may begin judging before the official start date.

Checkpoint: Preliminary Results Announced

Phase 3: Escalation Period

Purpose: Sometimes a Watson will want to flag something they believe the Lead Judge got wrong.

• After the preliminary results are announced, Watsons have 24 hours to flag any issues for re-review

• Watsons must pay with their Signal Score to flag an issue

• Watsons can then leave an extra comment explaining their rationale

Phase 4: Final Judgments

Purpose: The Lead Judge will re-review any issues flagged during the Escalation Period. The Sherlock Judge will supervise and interject if needed.

• The Lead Judge gets (# of controversial issues / 20) days to complete Phase 4, with a minimum of 2 days.

• Lead Judge will review all controversial issues (new ones from the Escalation Period and old ones that he already judged)

• Lead Judge will make final decisions on all controversial issues

• The Sherlock Judge will closely monitor the decisions during Phase 4, be available for any questions, ask questions if something is unclear, and only overturn a decision if it’s clearly wrong.

Checkpoint: Final Results Announced