Quiz check: fix before handing over, with an independent second review
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 29s

- The check can rewrite a whole question (garbled or contradictory stems,
  missing data, two defensible answers); a key fix that changes nothing is
  skipped.
- Removed questions are replaced with new ones, so a 50-question quiz stays 50.
- A second, independent model call reads the whole quiz for badly written
  items, helped by a scan for leftovers such as "A 78 kg? No - an 80 kg".
- The quiz is checked again after any rewrite, replacement or open problem,
  up to three rounds, before it is handed over.
- Reviewers see full rationales (a shortened one is marked as shortened).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Elijah 2026-11-16 10:07:32 -07:00
parent 85d5c9186d
commit 1a212bb8ba
8 changed files with 412 additions and 26 deletions

View file

@ -78,9 +78,23 @@ After a quiz is generated (unless "Check answers" is off), the generation model
as Python data plus the request (count, select-all share, calculation allowance). In at most
three Python runs it checks the count, question types, number of correct answers per type,
numeric answers, the calculation allowance and topic breadth, then reports with `submit_review`.
Fixes are applied only when the question stays valid. Remarks the model files as cosmetic or
informational (answer position, quotes) are not counted as failed checks; answer order is
shuffled whenever a quiz is taken.
Fixes are applied only when the question stays valid: a new key (`set_correct`, skipped when it
changes nothing), a numeric answer, a type or topic, a complete `rewrite` of the question (for a
garbled or contradictory stem, missing data, two defensible answers), or `remove`. Remarks the
model files as cosmetic or informational (answer position, quotes) are not counted as failed
checks; answer order is shuffled whenever a quiz is taken.
Then a **second review**: a fresh model call with no Python reads the whole quiz as a careful
student and item writer would, and rewrites or removes questions that are badly written (a
writer changing its mind mid-stem such as "A 78 kg? No — an 80 kg client", contradictions,
missing data, two defensible answers, a rationale that disagrees with the key). A text scan
for leftovers ("? No", "Actually,", "wait") is passed to it as hints (`suspiciousWording`).
The quiz is not handed over while anything is outstanding: removed questions are replaced with
new ones, and after any rewrite, replacement, failed check or refused fix the whole quiz is
checked again (both reviews), up to three rounds (`VERIFY_ROUNDS` in `server/ai/generate.ts`).
A count left short after the last round is still refilled. The notes say how many rounds ran
and list every fix by round; anything still failing after three rounds is listed as failed.
## Quiz generation rules