# Judging the Hermes Agent Plugin Hackathon

For the orchestrator. Two files sit beside this one: `field.json`, every entry (1,464 plugins, each a pull request on github.com/NousResearch/hermes-agent with its repository and pinned commit) plus the two reference plugins, and `score.py`, Python 3 with the standard library only. Get both into your workspace before anything else, from the same address as this file:

    curl -LsSf https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/field.json -o field.json && curl -LsSf https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/score.py -o score.py

`GOAL_ID` below is this goal's id, from `.imd/reads/goal.json`. Each step of `score.py` reads the goal's results from https://api.imd.fun itself, so it needs no file from an earlier episode, and it writes to standard out one JSON object, `{"spawn": [...]}`: the batches for your decisions file's `spawn`, as written. Do not edit their briefs or items. Wake on every batch you spawn.

## The four rounds, in order

1. **Round 1, episode 1.** `python3 score.py read1`. One batch, `read1`: 147 items, every plugin read by one seat, and every seat also scores the two reference plugins, so a hard grader can be told from a weak batch.
2. **Round 2, when `read1` has settled.** If any `read1` items failed, first spawn them again once, `{"name": "read1-again", "itemsFrom": {"batch": "read1", "items": "failed"}}` with read1's brief, output and merge, and wait for it. Then `python3 score.py read2 GOAL_ID`: 24 items in batches of five, the top 240 read again by fresh seats.
3. **Round 3, when those have settled.** `python3 score.py deep GOAL_ID`: 60 items in ten batches of six. The top 120 by their reads, each run inside real Hermes and through the catalog's own check by three seats (RUN.md, tryit.py), and each seat ranks the plugins it ran.
   - **Plugins with no first read.** One round 1 seat (read1-083) never answered, so its ten plugins have no read, and eleven more have none a seat could use. At your first wake after round 3 is dealt, run `python3 score.py late GOAL_ID` and spawn what it writes: two small batches, `read1-late` and `read1-late2`, which give each such plugin two fresh seats. It writes them once. Wait for nothing else because of them.
   - **Before the final.** At the wake where round 3 is far enough along for `final` (the straggler rule below), first run `python3 score.py extra GOAL_ID`. It gives round 3 seats to the plugins a hung seat held back: any that stands in the top 120 but is in no round 3 seat, and any that is owed a second read (it has one read, at or above round 2's cut, because seat read2-021 never answered). Three seats each, beside plugins already in round 3. If it writes no batch, run `final` in the same episode. If it writes batches (`deep-x`, and `deep-y` when there are more than five items; ten items at most), spawn them, wake on them, and run `final` once they have settled or run 60 minutes. If it says it is too early, run it again at the next wake. Do not wait for the late batches (`read1-late`, `read1-late2`) beyond 40 minutes before running it.
4. **Round 4, when those have settled.** `python3 score.py final GOAL_ID`: twelve items in three batches. Twelve seats rank the 20 finalists.
5. **Then** `python3 score.py done GOAL_ID` writes the final order as a table. Put the table in your notes and finish.

**Do not hold a round for stragglers.** When you wake and a batch still has items running, count: once nine in ten of its items are accepted and the rest have run over 30 minutes, go on to the next step at once (round 3 is the exception: wait for 55 of its 60, or 150 minutes; round 4 needs 10 of its 12). `score.py` reads whatever has landed, and a result that lands late still counts in every later step, so nothing is lost by going on. Wake on the batches you spawned or after 40 minutes, whichever comes first.

In every round, a batch whose items failed may be run again once with `itemsFrom`, as in step 2. Put each step's line from standard error in your notes (plugins read, read noise, grading spread) with the batch names you spawned. If `score.py` cannot reach api.imd.fun from your machine, spawn nothing and say so in your notes: the next episode runs on another seat.

## Why it is built this way

One read per plugin is enough to find the top few hundred, once each seat's grading is measured against the same two reference plugins (done-bell, a small Desktop notifier, and Home Assistant, a full platform plugin). The top 240 get a second read, the top 120 three more from seats that run them inside Hermes and rank what they ran, and the last 20 are ranked by twelve seats that each see all of them. Every step is seeded, so it gives the same answer when run again.

## Changed during the run

Round 1's results were measured before round 2 was dealt (2026-11-11 22:50Z, 145 of 147 seats in), and three things were changed for the rounds not yet started (a fourth, at 23:14Z: rounds no longer wait for the last few slow items, after two round 1 seats ran past 45 minutes while 145 had finished in under 20). They were decided from the whole field's numbers, without looking at how any one entry would move.

1. **Every judge family is graded on one curve.** Plugins are dealt to seats at random, so Claude seats and Codex seats read the same mix. They did not score it the same: 20.7% of Codex reads gave 27 or more of 30, against 10.9% of Claude reads, and 4.6% against 0.6% at 29 or more. A total now counts for what all judges gave at the same standing within its own family (`curve` in score.py; a Claude 27 is worth 27.8 and a Codex 27 is 26.6). Each seat's own grading is still corrected by the two reference plugins after that.
2. **The rounds are wider.** Totals bunch at the top (245 plugins at 27 or more after one read), so the second read takes the top 240 (was 200 plus seats' "top ten" marks, which a quarter of all entries got), round 3 the top 120 (was 90) and the final 20 (was 15) with twelve seats (was ten).
3. **Round 3 seats rank what they ran.** A score out of 30 cannot separate plugins that all score 28 to 30, so each round 3 seat also ranks its six, and a plugin's round 3 result is half its score with seat grading taken out and half its head-to-head standing across the three seats that ranked it (Bradley-Terry). The final round was already a ranking.
4. **One read is trusted as far as re-reading shows it can be** (23:31Z, before round 3 was dealt). A fresh second read of the top 240 kept 64% of their first read's lead over the field's mean, so a plugin's standing is its reads' mean pulled toward the field's mean by that measure (`two_reads`), where before the pull came from the two reference plugins, which judges score far more alike than they score entrants. It matters for plugins with one read against plugins with two.
5. **No plugin goes unjudged because a seat hung** (00:12Z, round 3 running). Seat read1-083 had held ten plugins for two hours without answering. `score.py late` gives every plugin without a usable first read two fresh seats, and `score.py extra` gives any of them that then stands in the top 120 its three round 3 seats before the final is dealt. Tested on simulated fields where the hung seat held the two best plugins.
6. **A plugin owed a second read gets round 3 instead** (00:56Z, before the final was dealt). Seat read2-021 never answered, so the plugins it held stood on one read against plugins with two, and so did the plugins of the two round 1 seats that answered after round 2 was dealt. Ten of them had a first read at or above round 2's cut. `score.py extra` now gives each of those its three round 3 seats, beside plugins already in round 3, where the hands-on round and the seats' rankings decide.