Keep rationales on what the question tests
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 30s

Med math rationales were padded with unrelated drug facts (TSH and heart
rate for a levothyroxine tablet calculation) because the math prompt
asked for "one safety check" and the rationale format asked for at
least two bullets.
- Rationales take 1 to 4 bullets, each about what the question tests,
  with fewer bullets preferred to padding.
- Med math rationales explain only the calculation: the conversion, the
  method and an arithmetic check.
- The second review rewrites a rationale or "why" that brings in facts
  the question does not test.
- The outside-chatbot instructions say the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Elijah 2026-11-16 15:33:38 -07:00
parent 0ad6290250
commit fe94dbf876
4 changed files with 11 additions and 6 deletions

View file

@ -109,9 +109,13 @@ Select-all items have exactly 6 choices with 2 to 5 correct, most often 3 or 4.
appear in the batch prompt, the planner, the review and proofread rewrites, "improve questions"
and the outside-chatbot instructions.
**Rationales** are asked for as Markdown: one bold key sentence, then 2 to 4 short bullet points,
**Rationales** are asked for as Markdown: one bold key sentence, then 1 to 4 short bullet points,
without repeating the per-choice "why" (`RATIONALE_FORMAT`, also in the question schema, the
review's rewrite schema, "improve questions" and the outside-chatbot instructions). The web app
review's rewrite schema, "improve questions" and the outside-chatbot instructions). Every bullet
must be about what the question tests; fewer bullets are preferred to padding with monitoring or
side effects. Med math rationales explain only the calculation (conversion, method and an
arithmetic check), and the second review rewrites a rationale that brings in facts the question
does not test (such as TSH monitoring in a levothyroxine tablet calculation). The web app
lays out older one-paragraph rationales the same way when it shows them (`formatRationale`).
Then a **second review**: a fresh model call with no Python reads the whole quiz as a careful