Quiz check: fix before handing over, with an independent second review
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 29s
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 29s
- The check can rewrite a whole question (garbled or contradictory stems, missing data, two defensible answers); a key fix that changes nothing is skipped. - Removed questions are replaced with new ones, so a 50-question quiz stays 50. - A second, independent model call reads the whole quiz for badly written items, helped by a scan for leftovers such as "A 78 kg? No - an 80 kg". - The quiz is checked again after any rewrite, replacement or open problem, up to three rounds, before it is handed over. - Reviewers see full rationales (a shortened one is marked as shortened). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
85d5c9186d
commit
1a212bb8ba
8 changed files with 412 additions and 26 deletions
20
docs/AI.md
20
docs/AI.md
|
|
@ -78,9 +78,23 @@ After a quiz is generated (unless "Check answers" is off), the generation model
|
|||
as Python data plus the request (count, select-all share, calculation allowance). In at most
|
||||
three Python runs it checks the count, question types, number of correct answers per type,
|
||||
numeric answers, the calculation allowance and topic breadth, then reports with `submit_review`.
|
||||
Fixes are applied only when the question stays valid. Remarks the model files as cosmetic or
|
||||
informational (answer position, quotes) are not counted as failed checks; answer order is
|
||||
shuffled whenever a quiz is taken.
|
||||
Fixes are applied only when the question stays valid: a new key (`set_correct`, skipped when it
|
||||
changes nothing), a numeric answer, a type or topic, a complete `rewrite` of the question (for a
|
||||
garbled or contradictory stem, missing data, two defensible answers), or `remove`. Remarks the
|
||||
model files as cosmetic or informational (answer position, quotes) are not counted as failed
|
||||
checks; answer order is shuffled whenever a quiz is taken.
|
||||
|
||||
Then a **second review**: a fresh model call with no Python reads the whole quiz as a careful
|
||||
student and item writer would, and rewrites or removes questions that are badly written (a
|
||||
writer changing its mind mid-stem such as "A 78 kg? No — an 80 kg client", contradictions,
|
||||
missing data, two defensible answers, a rationale that disagrees with the key). A text scan
|
||||
for leftovers ("? No", "Actually,", "wait") is passed to it as hints (`suspiciousWording`).
|
||||
|
||||
The quiz is not handed over while anything is outstanding: removed questions are replaced with
|
||||
new ones, and after any rewrite, replacement, failed check or refused fix the whole quiz is
|
||||
checked again (both reviews), up to three rounds (`VERIFY_ROUNDS` in `server/ai/generate.ts`).
|
||||
A count left short after the last round is still refilled. The notes say how many rounds ran
|
||||
and list every fix by round; anything still failing after three rounds is listed as failed.
|
||||
|
||||
## Quiz generation rules
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue