Faster quiz generation: planned parallel batches, room to reason
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 25s
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 25s
- Output limits leave room for reasoning (answer plus 8k-64k by effort, up to the model maximum); a call cut off before answering is retried with double the limit. At "max" effort, batches were being cut off at 16k tokens. - Quizzes are planned first (one distinct angle per question), then written in waves of four batches; med math runs alongside the first wave. - Replacements for removed questions are planned and written in parallel. - The Python check and the second review run at the same time each round. Measured live (DeepSeek flash, max reasoning, 50 questions): 7 min 21 s with no duplicates, against 20 min and 29 replaced duplicates without planning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1a212bb8ba
commit
2454ab6e95
8 changed files with 213 additions and 51 deletions
|
|
@ -169,7 +169,8 @@ describe('the quiz check', () => {
|
|||
const proofreads = chatCalls().filter((r) => r.body.tool_choice?.function?.name === 'submit_proofread');
|
||||
expect(proofreads.length).toBeGreaterThanOrEqual(2);
|
||||
// The leftover was pointed out to the reviewer, and the question no longer has it.
|
||||
expect(JSON.stringify(proofreads[0].body.messages)).toMatch(/looks like a leftover/);
|
||||
// (The first second-review runs alongside the check that planted it, so a later one sees it.)
|
||||
expect(proofreads.some((p) => /looks like a leftover/.test(JSON.stringify(p.body.messages)))).toBe(true);
|
||||
expect(draft.payload.questions.some((q: any) => /\?\s*No\b/.test(q.stem))).toBe(false);
|
||||
expect(draft.meta.notes.join(' | ')).toMatch(/Second review: Question \d+: rewritten/);
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue