Participating in Judging Contests
So you want to be a Sherlock judge...
5% of the value of the audit contest pot will be publicly available. If an audit has a $50k pot, then the portion of the “judging pot” that will be publicly available is $2,500.
The Lead Judge will also receive 5% of the audit contest pot. Here is how you can become a Lead Judge.
Immediately after an audit contest ends, the judging contest will begin. The judging contest will be contingent on the number of issues. More specifically, one day will be added per 75 issues submitted in the contest (200 issues = 3-day judging contest, 250 issues = 4-day judging contest).
- 2.Sign up as a Watson if you haven't already.
- 3.Find active contests in the
JUDGING CONTESTSsection of the Contests page, or find upcoming ones by looking for a "Judge Contest" button on a protocol's detailed contest page - 4.Click on that contest then click the "Judge Contest" button on the lower right side.
- 5.A new private repo will be created for you in GitHub.
- 6.The repo will contain a Markdown (.md) file with each submitted issue. Your job is to sort the Markdown files. See the next section.
DONT's
- Change file content
- Remove files (only move between folders)
- Rename files (has a single exception)
- Create folders that are not formatted either ###-M or ###-H
### stands for any number, we would suggest to start with 001 and increase sequentially (002, 003, 004)
By default every {number}.md file in the root of the repo is considered an unjudged issue.
You've already completed the steps above and now you're ready to start judging:
For each Markdown (.md) file, you will:
Sherlock judges low, info and false issues as invalid
1. Decide the severity (High/Medium/Invalid)
- If High, create a new folder with a 3 digit number and an “-H” (ex. 001-H) and put the corresponding .md file in the folder
- If Medium, create a new folder with a 3 digit number and an “-M” (ex. 001-M) and put the corresponding .md file in the folder
- If Invalid, keep the .md file in the root (or create a “invalid” folder to keep track of them yourself)
An example High severity issue might look like this:
Individual issue
001-H/
- 003.md
2. Decide if the issue is a duplicate or not
- If it’s a duplicate, put it in the same folder as an issue that it’s a duplicate of
- If it’s not a duplicate, it will live in its own folder (if it’s a High or Medium)
A family of duplicate High issues might look like:
Family of issues
001-H/
- 003.md
- 007.md
- 049.md
3. If there are duplicates, pick the “best” issue out of the bunch (the issue that describes the vulnerability best)
For example, if 7.md was the “best” issue, you would rename it like so:
Family of issues (with best issue labeled)
001-H/
- 003.md
- 007-best.md
- 049.md
Once you’ve gone through all of the issues, every issue should be in a folder. The top-level folder structure might look like:
001-H/
002-H/
001-M/
002-M/
003-M/
This would mean that there are 2 High severity vulnerabilities, and 3 Medium severity vulnerabilities.
If you were to open up all the folders above, it might look like this:
001-H/
- 003.md
- 005-best.md
- 012.md
002-H/
- 006.md
001-M/
- 007.md
- 013-best.md
002-M/
- 011-best.md
- 016.md
003-M/
- 014.md
- 001.md
- 002.md
- 004.md
- 008.md
- 009.md
- 010.md
- 015.md
This structure still means that there are 2 High severity vulnerabilities, and 3 Medium severity vulnerabilities (just like the top-level folder structure above).
"-best" only needs to be added when there is more than 1 .md file in a folder.
Once the issues have been sorted into folders and "best" reports labeled, you are done!
In order to become eligible for a payout, you need to submit more true positives (
invalid, medium or high) than there are invalid issues* in the final result of the contest. Basically, if judging is a multiple-choice test ('A' or 'B' or 'C' or 'D'), you need to outperform someone who chooses 'C' for every answer on the test. * Will be
medium (or high) in case there are more medium (or high) issues in the final result than invalid issues- 1.Severity (Medium/High) = 1 point
- 2.Duplication (Is it in a folder with at least 1 correct duplicate issue? Or correctly put in a solo folder?) = 1 point
- 3.Best issue (Was the best issue of the duplicate family correctly chosen?) = 1 point
If there were 80
invalid, 10 medium, 10 high issues in a contest, then a contestant would need to submit at least 81 true positives in order to become eligible for a payout.By default all the issues will be invalid (80 true positives, 20 false positives in the
invalid category), picking 1 valid medium or high will allow you to exceed the threshold and make you eligible for a payout.Rewards only count for valid
medium or high issues, invalid issues are excluded from the calculation.The formula for payouts is very similar to the sybil-resistant formula for audit contests.
Remember that in an audit contest, you get paid way more if you find an issue that no one else finds. The dynamic is the same in a judging contest.
In the judging contest, you get paid way more if you get something correct that no one else gets correct. And you get paid less if 4 people get it correct vs. 2 people get it correct, etc.
There’s also a twist:
- 1.Correct "Severity" judgments are worth 10x
- 2.Correct "Duplicate" judgments are worth 5x
- 3.Correct “Best Report” judgments are worth 1x
And the judging contest uses an even stricter sybil formula, which means getting a “solo correct” issue is worth A LOT more than getting an issue correct alongside 3 other people. Sherlock hopes this will reimburse judges who spend the time to understand the most difficult issues in the judging contest.
The sybil-resistant formula is:
question_weight / 2.01 ^ (X - 1)
Where question_weight is the multiplier for the question type (Severity, Duplication, Best), and X represents the number of judges who got the question correct.
After this, each judge will receive an individual score for their performance. And based on the individual_score / total_score, a proportional percentage of the contest pot will be awarded.
For example, if the contest pot is $2,500 and a judge receives 50% of all the points, then that judge will receive $1,250.
Last modified 2mo ago