Varied question openers, readable rationales, plain practice score
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 30s
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 30s
- Stems no longer all open the same way: the writing rules, batch prompt and planner ask for varied openings (age openings stay part of the mix), and when one opener covers more than a quarter of a quiz the excess stems have only their opening reworded, keeping every number, before the check. - Rationales are asked for as Markdown (a bold key point, then short bullets); older one-paragraph rationales are laid out the same way on display. - The practice score in the quiz header is a plain "x/y correct", with no percentage or colour. - Fix a flaky assertion: the mock's answer-key fix is a no-op when the shuffle already put the answer in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
10be5c39da
commit
517278be71
15 changed files with 382 additions and 24 deletions
|
|
@ -42,6 +42,7 @@ import { weakTopics } from '../services/stats.ts';
|
|||
import { getSetting, setSetting } from '../settings.ts';
|
||||
import { newId, now, parseJson } from '../util.ts';
|
||||
import { generateStructured, outputBudget, resolveModel, runChat, runChatWithMcp } from './service.ts';
|
||||
import { openerAdvice, repeatedOpeners, varyOpeners } from './openers.ts';
|
||||
import { applyQuizFixes, proofreadQuiz, verifyQuizDraft, type QuizCheck } from './verify.ts';
|
||||
|
||||
/** Most quiz checks per generation: the first, then re-checks after fixes and replacements. */
|
||||
|
|
@ -321,6 +322,10 @@ export type GenerateRequest = z.infer<typeof generateRequestSchema>;
|
|||
|
||||
/* ------------------------------ prompts ------------------------------ */
|
||||
|
||||
/** How a rationale is laid out, so it reads at a glance instead of as one block of text. */
|
||||
const RATIONALE_FORMAT =
|
||||
'Format the rationale in Markdown for quick reading: one bold sentence with the key point, then 2 to 4 short bullet points (one idea each, such as the principle or framework, the physiology or drug action, the safety point). No long paragraphs, and do not repeat the choices\' "why" text, which is shown beside each choice.';
|
||||
|
||||
const QUESTION_SCHEMA = {
|
||||
type: 'object',
|
||||
properties: {
|
||||
|
|
@ -348,7 +353,7 @@ const QUESTION_SCHEMA = {
|
|||
required: ['text', 'correct', 'why'],
|
||||
},
|
||||
},
|
||||
rationale: { type: 'string', description: 'Teaching explanation of the correct answer. Markdown allowed.' },
|
||||
rationale: { type: 'string', description: `Teaching explanation of the correct answer. ${RATIONALE_FORMAT}` },
|
||||
topic: {
|
||||
type: 'string',
|
||||
description: 'Broad topic: the main condition, drug class or body system in 1-3 words (e.g. "Stroke", "Heart failure", "Anticoagulants"). Never a subtype or aspect such as "Hemorrhagic stroke" or "Heparin client teaching". Reuse the same label for every question on that subject.',
|
||||
|
|
@ -394,13 +399,15 @@ const NUMERIC_RULE = `- Only write calculation items when the source material it
|
|||
|
||||
const NCLEX_RULES = `Write NCLEX-style (Next Generation NCLEX) items:
|
||||
- Test clinical judgment at the application and analysis level: recognise cues, analyse, prioritise, choose and evaluate actions. Avoid pure recall.
|
||||
- Open with a brief, realistic client scenario containing only relevant data (age, diagnosis, key findings, vital signs or lab values with units).
|
||||
- Give a brief, realistic client scenario containing only relevant data (age, diagnosis, key findings, vital signs or lab values with units).
|
||||
- Vary how stems open, as a real exam does: with the client ("A 68-year-old client…", "A client with…"), the nurse's situation ("The nurse is caring for…", "After receiving report…"), the setting or role ("In the emergency department…", "The charge nurse…", "During discharge teaching…"), a finding, order or report ("A client with heart failure reports…", "The health care provider prescribes…"), or the question itself. Mix these so no single opening dominates the quiz.
|
||||
- Ask one clear question. Use priority wording where it fits (first, best, most important, requires immediate follow-up) and build it on a real framework: ABCs, Maslow, the nursing process, acute before chronic, unstable before stable, safety, least invasive first.
|
||||
- Cover the client-needs areas where the material allows: safe and effective care, pharmacology, physiological adaptation, reduction of risk, health promotion, psychosocial integrity, delegation and assignment.
|
||||
- Every distractor must be plausible to a student with partial knowledge and wrong for a specific reason. Keep choices parallel in grammar and similar in length. Never use "all of the above" or "none of the above". Do not give the answer away through wording or length.
|
||||
- Multiple-choice items have exactly 4 choices with exactly one best answer. Select-all-that-apply items have 5 or 6 choices with 2 to 5 correct, and the stem ends with "Select all that apply."
|
||||
- Use generic drug names. Use current, standard practice.
|
||||
- The rationale teaches: state why the correct answer is correct, name the principle behind it, and give each choice its own "why".
|
||||
- ${RATIONALE_FORMAT}
|
||||
${NUMERIC_RULE}`;
|
||||
|
||||
|
||||
|
|
@ -928,6 +935,8 @@ async function runGeneration(app: AppContext, draftId: string, req: GenerateRequ
|
|||
: `The material involves calculations: you may include up to ${numericLeft} calculation question${numericLeft === 1 ? '' : 's'} (type "numeric", the student types the number). Do not exceed that.`,
|
||||
`Topic labels must be broad (the main condition, drug class or body system, 1-3 words), not subtypes or aspects.${topicsSoFar.length ? ` Reuse these existing topics wherever they fit: ${topicsSoFar.slice(0, 40).join(', ')}.` : ''}`,
|
||||
difficulty === 'mixed' ? 'Mix difficulty: mostly medium, with a few easy and a few hard.' : `Difficulty: ${difficulty}.`,
|
||||
'Vary how the stems open (the client and their age, the nurse\'s situation, the setting, a finding or order, or the question itself) so no single opening dominates.',
|
||||
draft.kind === 'quiz' ? openerAdvice(draft.questions) : '',
|
||||
sources.groups.length > 0 ? 'Spread the questions across the excerpts instead of clustering on one. Set "source" to the id of the excerpt each question is based on.' : '',
|
||||
req.source.deckId ? 'Turn the flashcards into questions: use each card\'s answer as the correct choice and write distractors that a student could confuse with it.' : '',
|
||||
]
|
||||
|
|
@ -995,7 +1004,7 @@ async function runGeneration(app: AppContext, draftId: string, req: GenerateRequ
|
|||
task: 'plan',
|
||||
purpose: 'planning questions',
|
||||
modelId: req.modelId,
|
||||
system: `${baseSystem(db, req, hasSources)}\n\nYou plan NCLEX-style quizzes. List distinct angles, one per question: each names the client situation and the single decision or fact it tests, in one line (for example "Continuous IV heparin, aPTT 3x control with gum bleeding: first action"). No two angles may test the same decision, fact or teaching point, even in a different scenario. Spread them over the client-needs areas the material supports (pharmacology, safety, prioritisation, delegation, teaching, lab values, complications, psychosocial). When the material is short, vary the situations and the area tested rather than repeating its points.`,
|
||||
system: `${baseSystem(db, req, hasSources)}\n\nYou plan NCLEX-style quizzes. List distinct angles, one per question: each names the client situation and the single decision or fact it tests, in one line (for example "Continuous IV heparin, aPTT 3x control with gum bleeding: first action"). No two angles may test the same decision, fact or teaching point, even in a different scenario. Vary the framing too: the setting (unit, clinic, home, emergency department), the nurse's role (bedside, charge nurse, delegating, teaching) and what prompts the question (a finding, a report, an order, a call light). Spread them over the client-needs areas the material supports (pharmacology, safety, prioritisation, delegation, teaching, lab values, complications, psychosocial). When the material is short, vary the situations and the area tested rather than repeating its points.`,
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
|
|
@ -1109,6 +1118,12 @@ async function runGeneration(app: AppContext, draftId: string, req: GenerateRequ
|
|||
draft.questions.map((q) => ({ ...q, topic: mapping.get(q.topic) ?? q.topic })),
|
||||
knownTopics.filter((t) => t.trim().split(/\s+/).length <= 2),
|
||||
);
|
||||
// When most stems open the same way ("A 72-year-old client…"), reword the openings of the
|
||||
// excess ones (stem only) before the check reads them.
|
||||
if (repeatedOpeners(draft.questions).length) setProgress(db, draftId, 0.95, 'Varying how questions open', draft);
|
||||
const varied = await varyOpeners(app, draft.questions, { modelId: req.modelId, signal });
|
||||
draft.questions = varied.questions;
|
||||
if (varied.asked > 0) notes.push(`Reworded how ${varied.reworded} question${varied.reworded === 1 ? '' : 's'} open${varied.reworded === 1 ? 's' : ''}, so fewer start the same way.`);
|
||||
}
|
||||
|
||||
// Check counts, the type mix, answer keys and calculations with Python, and keep fixing until
|
||||
|
|
@ -1404,9 +1419,9 @@ export function startImprove(app: AppContext, input: z.input<typeof improveReque
|
|||
const ids: string[] = [];
|
||||
const system = `You are an expert nursing educator and item writer reviewing existing exam questions.
|
||||
Improve each question while keeping what it tests and its correct answer(s) unless the keyed answer is factually wrong (if so, fix it and say so at the start of the rationale).
|
||||
- Make the stem clear, complete and free of clues; fix grammar.
|
||||
- Make the stem clear, complete and free of clues; fix grammar. Vary how the stems open so they do not all start the same way.
|
||||
- Replace weak or implausible distractors; keep choices parallel and similar in length; remove "all/none of the above".
|
||||
- Write a teaching rationale, and give every choice a one-sentence "why".
|
||||
- Write a teaching rationale, and give every choice a one-sentence "why". ${RATIONALE_FORMAT}
|
||||
- Keep the same question type unless it is clearly wrong for the content. Set a short topic label and a difficulty.
|
||||
Return the questions in the same order, one output question per input question.${req.instructions ? `\n\nAdditional instructions: ${req.instructions}` : ''}`;
|
||||
for (let i = 0; i < questions.length; i += 8) {
|
||||
|
|
|
|||
156
server/ai/openers.ts
Normal file
156
server/ai/openers.ts
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
/**
|
||||
* Variety in how quiz stems open. Models tend to start nearly every NCLEX item the same way
|
||||
* ("A 72-year-old client…"), which reads as monotonous and cues the student. After writing, the
|
||||
* openers are counted; when one dominates, the excess stems are reworded (stem only: options,
|
||||
* answer and rationale stay) before the quiz is checked.
|
||||
*/
|
||||
import type { DraftQuestion } from '../../shared/content.ts';
|
||||
import type { AppContext } from '../http.ts';
|
||||
import { generateStructured, outputBudget, resolveModel } from './service.ts';
|
||||
import { AIError } from './types.ts';
|
||||
|
||||
/** The first two words of a stem, lowercased, with ages and other numbers folded ("a #-year-old"). */
|
||||
export function stemOpener(stem: string): string {
|
||||
return stem
|
||||
.replace(/[*_`>#[\]()]/g, ' ')
|
||||
.toLowerCase()
|
||||
.replace(/\b\d+(?:\.\d+)?\s*-?\s*(?:years?|months?|weeks?|days?)\s*-?\s*old\b/g, '#-year-old')
|
||||
.replace(/\d+(?:\.\d+)?/g, '#')
|
||||
.split(/\s+/)
|
||||
.filter(Boolean)
|
||||
.slice(0, 2)
|
||||
.join(' ');
|
||||
}
|
||||
|
||||
/** Openers used by more than a quarter of the questions (at least 4), and which of them to reword. */
|
||||
export function repeatedOpeners(questions: Pick<DraftQuestion, 'stem' | 'type'>[], share = 0.25): { opener: string; example: string; total: number; reword: number[] }[] {
|
||||
if (questions.length < 8) return [];
|
||||
const allow = Math.max(3, Math.floor(questions.length * share));
|
||||
const groups = new Map<string, number[]>();
|
||||
questions.forEach((q, i) => {
|
||||
const o = stemOpener(q.stem);
|
||||
if (o) groups.set(o, [...(groups.get(o) ?? []), i]);
|
||||
});
|
||||
return [...groups]
|
||||
.filter(([, list]) => list.length > allow)
|
||||
.map(([opener, list]) => {
|
||||
// Keep every k-th one as it is so the remaining uses are spread out; calculations keep theirs.
|
||||
const keepEvery = list.length / allow;
|
||||
const kept = new Set(Array.from({ length: allow }, (_, k) => list[Math.floor(k * keepEvery)]));
|
||||
const first = questions[list[0]!]!.stem.replace(/[*_`]/g, '').trim();
|
||||
return {
|
||||
opener,
|
||||
example: first.split(/\s+/).slice(0, 3).join(' '),
|
||||
total: list.length,
|
||||
reword: list.filter((i) => !kept.has(i) && questions[i]!.type !== 'numeric'),
|
||||
};
|
||||
})
|
||||
.filter((g) => g.reword.length > 0);
|
||||
}
|
||||
|
||||
/** Numbers in a text (ages, vital signs, doses, lab values), which a reworded stem must keep. */
|
||||
function numbersIn(text: string): string[] {
|
||||
return (text.match(/\d+(?:[.,]\d+)?/g) ?? []).map((n) => n.replace(',', '.'));
|
||||
}
|
||||
|
||||
/**
|
||||
* A reworded stem is accepted only when it keeps every number of the original, keeps the
|
||||
* "Select all that apply." ending of a multi item, and no longer opens the repeated way.
|
||||
*/
|
||||
export function acceptRewording(original: Pick<DraftQuestion, 'stem' | 'type'>, stem: string, repeated: Set<string>): string | null {
|
||||
let next = stem.trim();
|
||||
if (next.length < 20 || next.length > original.stem.length * 2 + 200) return null;
|
||||
if (repeated.has(stemOpener(next))) return null;
|
||||
const have = numbersIn(next);
|
||||
for (const n of numbersIn(original.stem)) {
|
||||
const at = have.indexOf(n);
|
||||
if (at < 0) return null;
|
||||
have.splice(at, 1);
|
||||
}
|
||||
if (original.type === 'multi' && !/select all that apply\.?\s*$/i.test(next)) next = `${next.replace(/\s+$/, '')} Select all that apply.`;
|
||||
return next;
|
||||
}
|
||||
|
||||
const OPENER_SYSTEM = `You edit NCLEX-style questions for variety. Too many stems in this quiz open the same way. Reword only the opening of each listed stem so the quiz reads like a real exam, where items open in many ways, for example:
|
||||
- with the nurse's situation: "The nurse is caring for…", "While reviewing the morning laboratory results, the nurse notes…", "After receiving report, the nurse…"
|
||||
- with the setting or role: "On a medical-surgical unit…", "In the emergency department…", "The charge nurse is making assignments…", "During discharge teaching…"
|
||||
- with a finding, order or report: "A client with heart failure reports…", "The health care provider prescribes…", "Two hours after surgery, a client's…"
|
||||
- or with the question itself when the scenario is short: "Which finding in a client receiving… requires immediate follow-up?"
|
||||
Openings with the client's age are fine elsewhere in the quiz; these stems just need a different opening from the repeated one.
|
||||
Keep everything the question tests: the same client facts, every number exactly (age, vital signs, lab values, doses), the same question asked, and an ending of "Select all that apply." where there is one. Include the age only where it is in the original, but it need not come first. Use "client", never "patient". Make the listed stems open differently from one another too. Call submit_stems once with every listed question.`;
|
||||
|
||||
/**
|
||||
* Rewords the stems of over-repeated openers. Returns the questions with the accepted rewordings
|
||||
* and how many were changed. Never throws except on cancel: the quiz is fine without it.
|
||||
*/
|
||||
export async function varyOpeners(
|
||||
app: AppContext,
|
||||
questions: DraftQuestion[],
|
||||
opts: { modelId?: string | null; signal?: AbortSignal },
|
||||
): Promise<{ questions: DraftQuestion[]; reworded: number; asked: number }> {
|
||||
const groups = repeatedOpeners(questions);
|
||||
const targets = groups.flatMap((g) => g.reword);
|
||||
if (targets.length === 0) return { questions, reworded: 0, asked: 0 };
|
||||
const repeated = new Set(groups.map((g) => g.opener));
|
||||
try {
|
||||
const result = await generateStructured(app, {
|
||||
task: 'write',
|
||||
purpose: 'varying question openers',
|
||||
modelId: opts.modelId,
|
||||
system: OPENER_SYSTEM,
|
||||
messages: [
|
||||
{
|
||||
role: 'user',
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text: `${groups.map((g) => `"${g.example}…" opens ${g.total} of the ${questions.length} stems.`).join(' ')}\n\nReword the opening of these stems:\n\n${targets
|
||||
.map((i) => `Q${i + 1}: ${questions[i]!.stem}\n(Correct answer: ${questions[i]!.options.filter((o) => o.correct).map((o) => o.text).join('; ')})`)
|
||||
.join('\n\n')}`,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
toolName: 'submit_stems',
|
||||
toolDescription: 'Submit the reworded stems.',
|
||||
schema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
stems: {
|
||||
type: 'array',
|
||||
items: { type: 'object', properties: { question: { type: 'integer', description: 'The question number (Qn).' }, stem: { type: 'string' } }, required: ['question', 'stem'] },
|
||||
},
|
||||
},
|
||||
required: ['stems'],
|
||||
},
|
||||
maxOutputTokens: outputBudget(resolveModel(app, 'write', opts.modelId), 1_000 + targets.length * 250),
|
||||
signal: opts.signal,
|
||||
});
|
||||
const list = (result.data as { stems?: unknown }).stems;
|
||||
const wanted = new Set(targets);
|
||||
const next = [...questions];
|
||||
let reworded = 0;
|
||||
for (const raw of Array.isArray(list) ? list : []) {
|
||||
const r = raw as { question?: unknown; stem?: unknown };
|
||||
const i = typeof r.question === 'number' ? r.question - 1 : -1;
|
||||
if (!wanted.has(i) || typeof r.stem !== 'string') continue;
|
||||
const stem = acceptRewording(questions[i]!, r.stem, repeated);
|
||||
if (!stem) continue;
|
||||
wanted.delete(i);
|
||||
next[i] = { ...questions[i]!, stem };
|
||||
reworded += 1;
|
||||
}
|
||||
return { questions: next, reworded, asked: targets.length };
|
||||
} catch (err) {
|
||||
if (opts.signal?.aborted || (err instanceof AIError && err.code === 'aborted')) throw err;
|
||||
app.log('warn', 'Varying question openers failed', err instanceof Error ? err.message : err);
|
||||
return { questions, reworded: 0, asked: targets.length };
|
||||
}
|
||||
}
|
||||
|
||||
/** A line for the writing prompt naming the openers already overused, so new questions avoid them. */
|
||||
export function openerAdvice(questions: Pick<DraftQuestion, 'stem' | 'type'>[]): string {
|
||||
const groups = repeatedOpeners(questions, 0.2);
|
||||
if (groups.length === 0) return '';
|
||||
return `Many questions so far open with ${groups.map((g) => `"${g.example}…"`).join(' or ')}: open yours in other ways.`;
|
||||
}
|
||||
|
|
@ -119,7 +119,7 @@ const SUBMIT_REVIEW_QUESTION_TEXT = {
|
|||
type: { type: 'string', enum: ['single', 'multi', 'numeric'] },
|
||||
stem: { type: 'string' },
|
||||
options: { type: 'array', items: { type: 'object', properties: { text: { type: 'string' }, correct: { type: 'boolean' }, why: { type: 'string' } }, required: ['text', 'correct'] } },
|
||||
rationale: { type: 'string' },
|
||||
rationale: { type: 'string', description: 'Markdown: one bold key sentence, then 2 to 4 short bullet points.' },
|
||||
},
|
||||
required: ['type', 'stem', 'options', 'rationale'],
|
||||
};
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue