Study/docs/AI.md
Elijah 1a212bb8ba
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 29s
Quiz check: fix before handing over, with an independent second review
- The check can rewrite a whole question (garbled or contradictory stems,
  missing data, two defensible answers); a key fix that changes nothing is
  skipped.
- Removed questions are replaced with new ones, so a 50-question quiz stays 50.
- A second, independent model call reads the whole quiz for badly written
  items, helped by a scan for leftovers such as "A 78 kg? No - an 80 kg".
- The quiz is checked again after any rewrite, replacement or open problem,
  up to three rounds, before it is handed over.
- Reviewers see full rationales (a shortened one is marked as shortened).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:07:32 -07:00

342 lines
23 KiB
Markdown

# AI connections and verification
Settings → AI manages provider connections and model routing. Add a connection, test it,
discover models where supported (or add a custom model), enable models and select defaults
for generation, chat, explanations and vision. Output limits, capabilities, pricing and
reasoning settings can be edited per model. Catalog prices are estimates, not billing records.
Adapters use OpenAI Responses, Anthropic Messages, Gemini generateContent, and Chat
Completions for DeepSeek, OpenRouter and OpenAI-compatible servers. Ollama is connected as an
OpenAI-compatible server at `http://<host>:11434/v1` (migration 6 moved existing Ollama
connections there).
OpenRouter (`https://openrouter.ai/api/v1`) needs a key; its test calls `/key`, so a bad key
fails even though the model list is public. Discovery reads each model's context length,
output limit, prices (with cache reads), tool and reasoning support and image input from
`/models`. Reasoning effort is sent as `reasoning: { effort }`, and requests carry
`X-Title: Study Desk`.
Reasoning effort has seven levels (Off, Minimal, Low, Medium, High, Extra high, Max); each
adapter maps them onto the nearest value its API accepts. DeepSeek accepts low / high / max
and turns thinking off at "Off". DeepSeek rejects forced tool calls in thinking mode, so
structured generation keeps thinking on and asks for the tool call in the prompt; answer keys
were noticeably less reliable with thinking off. Its `reasoning_content` is replayed between
tool steps, as the API requires.
Cost estimates split input into fresh, cache-hit and cache-write tokens. A model's "Cached
input price" is used when set; otherwise the provider's published discount applies (10% of
input for GPT-5+, Claude and Gemini; DeepSeek's catalog cache-hit price). DeepSeek's
off-peak half price is applied by time of call. Configure the endpoint as shown in Settings.
Local models are reached **from the server/container**: `localhost` inside Docker is the
container itself, not your workstation or Unraid host. Use a reachable LAN address instead.
Keys are encrypted with AES-256-GCM in SQLite using `/data/secret.key` or `APP_SECRET`.
Only the server decrypts them. Keep the key with backups; changing it breaks decryption.
Documents are sent to the selected provider when you request AI processing; extraction and
full-text indexing otherwise run locally. Review generated nursing content and source
references before using it for study.
## Reading scanned PDFs (OCR)
A PDF with little selectable text is read by the "Vision and OCR" model, which must accept
PDFs (OpenAI, Anthropic, Gemini; OpenAI-compatible servers when the model is marked "PDF").
Long files are split with pdf-lib into batches of 8 pages (smaller if a batch exceeds 24 MB)
and sent one batch per request, so there is no whole-file size cap. Each batch keeps its real
page numbers for citations. Finished batches are cached beside the upload
(`uploads/<file>.ocr.json`), so a restart or a failed batch does not repeat paid work; "Read
with AI" on the document forces a fresh read. A batch that fails twice is skipped with a
warning on the document. At most 600 pages are read per file.
## Semantic search
Settings → AI → Semantic search chooses an embeddings provider and model (suggested:
OpenAI `text-embedding-3-small`, Gemini `gemini-embedding-001`, or e.g. `nomic-embed-text` on
Ollama through an OpenAI-compatible connection; Anthropic, DeepSeek and OpenRouter connections
are not used for embeddings). Until something else is chosen, the built-in
model is used: BAAI bge-small-en-v1.5 (8-bit ONNX, 384 dimensions) running on the server's CPU
through transformers.js. The Docker image ships the model in `/app/models` (`LOCAL_MODELS_DIR`),
so it works without internet access; outside Docker it is downloaded once (about 35 MB) from
huggingface.co into `DATA_DIR/models`. It costs nothing. If the model cannot load, Settings shows
the actual error. `LOCAL_EMBEDDINGS=off` disables the default. Every document passage is embedded in the background and
stored as a vector in SQLite (`chunk_vectors`). Search, the assistant's document search and
focused generation then merge keyword (FTS5) and meaning-based rankings with reciprocal-rank
fusion, so exact terms still match while paraphrased questions find the right passage. If the
embedding provider fails, search silently falls back to keywords. Changing the model clears
and rebuilds the index. Costs appear under Usage as "semantic indexing" / "semantic search".
Quiz questions, flashcards and notes are still searched by keyword.
## Python sandbox and the quiz check
The assistant has a `run_python` tool, and generated quizzes are checked with it. Code runs in
Pyodide (CPython compiled to WebAssembly) inside a separate Node process started with
`--permission`. It can read only the Pyodide files, gets an empty environment, cannot reach
the network or start processes (blocked in Python and again in JavaScript), and has no access
to `/data`. One warm process is reused and recycled every 50 runs; a run that times out kills
it.
After a quiz is generated (unless "Check answers" is off), the generation model gets the draft
as Python data plus the request (count, select-all share, calculation allowance). In at most
three Python runs it checks the count, question types, number of correct answers per type,
numeric answers, the calculation allowance and topic breadth, then reports with `submit_review`.
Fixes are applied only when the question stays valid: a new key (`set_correct`, skipped when it
changes nothing), a numeric answer, a type or topic, a complete `rewrite` of the question (for a
garbled or contradictory stem, missing data, two defensible answers), or `remove`. Remarks the
model files as cosmetic or informational (answer position, quotes) are not counted as failed
checks; answer order is shuffled whenever a quiz is taken.
Then a **second review**: a fresh model call with no Python reads the whole quiz as a careful
student and item writer would, and rewrites or removes questions that are badly written (a
writer changing its mind mid-stem such as "A 78 kg? No — an 80 kg client", contradictions,
missing data, two defensible answers, a rationale that disagrees with the key). A text scan
for leftovers ("? No", "Actually,", "wait") is passed to it as hints (`suspiciousWording`).
The quiz is not handed over while anything is outstanding: removed questions are replaced with
new ones, and after any rewrite, replacement, failed check or refused fix the whole quiz is
checked again (both reviews), up to three rounds (`VERIFY_ROUNDS` in `server/ai/generate.ts`).
A count left short after the last round is still refilled. The notes say how many rounds ran
and list every fix by round; anything still failing after three rounds is listed as failed.
## Quiz generation rules
Every quiz is NCLEX style. "Practice exam" writes 50 questions. Calculation (numeric) questions
are written only when the source material involves calculations (dosage, drip rates, mg/kg,
intake and output...): up to 5 in an exam, about one in ten otherwise, none when the material
has no calculations. Topics are kept broad ("Stroke", not "Right hemisphere stroke"): the
prompt lists the class's existing topics, and narrower labels that contain an existing one are
merged into it after generation.
Flashcards are atomic: one fact per card, the back only the answer (a word, number or short
phrase), and no hints (the schema has no hint field any more). When generation finishes, one
small call names the set: a 2-6 word title for what it covers, different from the class's other
titles (a clash gets a number), and a one-sentence description. The generic title stays if
naming fails.
Malformed output: tool-call arguments that `jsonrepair` cannot fix are sent back to the model to
be rewritten as valid JSON (`repairStructured` in `server/ai/service.ts`); if that fails the
request is asked once more. A job that still fails on bad output or a passing provider error
starts over once with the same request ("Starting over"); a second failure fails the draft.
## MCP servers
Settings → AI → MCP servers connects Model Context Protocol servers over Streamable HTTP
(JSON-RPC over POST, answered as JSON or SSE). Headers (usually an `Authorization` token) are
encrypted like API keys and never returned to the browser. "Test and list tools" connects
afresh and lists the tools.
Each server has:
- **When to use it**: your instructions, added to the model's prompt next to the server's tools.
- **Where it can be used**: Assistant chat, Explain this, Generating material (quizzes,
flashcards, study guides), Quiz check and Share links, each switched separately. New servers
are on for chat and Explain only. "Share links" lets visitors of links with AI turned on use
the server through your account; leave it off for servers with private data or paid keys.
Tools are offered as `mcp_<server>__<tool>`, at most 48 per request. A server that cannot be
reached is skipped for that request, and the assistant is told it is unavailable. During
generation the model may make up to four rounds of tool calls before it must submit its
answer; the quiz check allows four external calls. Tool output is treated as data, not
instructions. On share links only servers with "Share links" switched on are offered.
### Signing in to MCP servers (OAuth)
Any server that answers 401 with a `WWW-Authenticate: Bearer resource_metadata=…` challenge
shows **Sign in** in its row. The app follows the MCP authorization spec (2025-06-18):
1. Reads the protected-resource metadata (RFC 9728), then the authorization server's
metadata (RFC 8414, falling back to OpenID discovery).
2. Registers itself by dynamic client registration (RFC 7591), unless a client ID (and
optional secret) is entered under the server's advanced settings. Registrations are kept
and reused.
3. Sends the browser to the sign-in page with PKCE S256, `state` and the `resource`
indicator (RFC 8707). The callback (`GET /api/ai/mcp/oauth/callback`) exchanges the code
and returns to the page that started it.
Tokens are encrypted like API keys, refreshed when within a minute of expiry, and retried
once after a 401. **Sign out** forgets the tokens. Pending sign-ins expire after 15 minutes.
### Built-in web search
Settings → Web search is a built-in entry backed by Parallel's MCP server
(`https://mcp.parallel.ai/mcp`, override with `WEB_SEARCH_URL`). It works without an account
on Parallel's free tier; **Sign in with Parallel** uses the same OAuth flow against
`/mcp?login` for higher limits, and then offers deep research (`web_research`) when switched
on. Only `web_search` and `web_fetch` (plus research) are offered; Parallel's bookkeeping
arguments (`session_id`, `client_model`) are hidden from the model, and ids, usage and
warnings are stripped from results. It is on for chat and Explain by default (off in tests),
with its own instructions and area switches like any server.
Tools load without a manual test: the server connects to every enabled MCP server (and the web
search) a few seconds after start, again every few minutes, and right after a server is saved
or signed in to. A listing that fails is retried once on a fresh connection before the server
is reported as unavailable for that turn.
The assistant never answers factual questions from its own knowledge. When the class materials
do not cover a question it asks "Search the web" or "No, skip it", and if the student skips it
says it has no information to answer. Without the web search it says so straight away. The
shared assistant answers only from the shared material (and web results, when allowed).
### Tool steps
Every tool call shows as one line: an icon for the kind of tool, a friendly name ("Searched
the web for: …", "Read pubmed.ncbi.nlm.nih.gov", "Drug reference: Lookup drug") and a
check or cross. While a step runs its icon breathes, and long steps report their stage
(generation and Python checking send "Reading your material", "Writing questions 1-8 of 15",
"Checking the questions with Python" and so on through `ToolActivity.stage`). Finished steps
read in the past tense ("Wrote 15 quiz questions"). Clicking it shows the pages found, the request and a clipped result.
More than three finished steps fold into "N steps". The model is told never to mention tool
names or internal ids. In a chat, the sliders button left of + turns individual servers (and
web search) off for that conversation; the choice is saved with it.
## Clarifying questions
The assistant's `ask_user` tool shows up to four questions as an inline form (choices, several
choices, text or a number, with defaults) and ends the turn. The answers are sent as the
user's next message, and the stored tool result travels with that message, so the model sees
question and answer together. The prompt tells the model to ask before generating when the
exam, materials or count are missing and not clear from the page in view, to ask at most once
per request, and to use defaults when the user says to go ahead. Every class is listed in the
prompt with its exams, so the exam question can offer them by name; the model is told never to
ask about question format (quizzes are always NCLEX style).
When web search is on in a chat and the class materials do not answer a question, the
assistant asks whether to search the web ("Search the web" / "Answer from general knowledge")
instead of answering from its own knowledge. It searches straight away only when asked to.
## Exams within a class
A class is divided into exams (or units, modules, weeks, topics or folders: Class settings →
"Divided into"). Exams are the existing folders; `classes.group_label` holds the name. The class
page shows one tab per exam plus "Not in an exam" and "All", opening on the last exam (the
choice is remembered per class). Uploads, pasted text, imports, new items and Generate default to
the exam in view. Generating for an exam uses only that exam's documents (also for weak areas
and searches), and the draft is saved in that exam; an exam from another class is refused. The
assistant is told the class's exams with their ids, which exam is in view, and passes `exam_id`
to its generate and search tools.
## Intelligent UI in chats
Assistant replies (owner, Explain and share-link chats) can include interactive blocks written
in OpenUI Lang inside ```` ```openui-lang ```` fences, rendered with `@openuidev/react-lang`
against the app's own component library (`shared/genui.ts` lists it; `web/src/genui/` renders
it): layout, callouts, stats, tables, steps, bar/line/pie charts with a table view, sliders and
inputs bound to variables, practice questions, flashcards and follow-up buttons that send a
message. The prompt lists the reply shapes that should become a block (comparisons, sequences,
calculations, safety points, a self-check after teaching), so blocks appear without being asked
for. Explain uses a reduced library without Question and Flashcards and is told to give
information only (`genUiPrompt('explain')`). Library telemetry is off (`OPENUI_TELEMETRY_DISABLED=1`, `DO_NOT_TRACK=1`).
## Medication math on request
Generate → "Include med math" with a count (default 5) asks for exactly that many dosage
calculation questions (`mathQuestions`), written at the chosen difficulty even when the
material has no calculations: easy is single-step doses and conversions, medium weight-based
doses and pump rates, hard drip rates, titrations and heparin protocols. They are written in
their own batches without the excerpts, then spread evenly through the quiz; the other batches
write no calculations. Without the option, calculations still appear only when the material
involves them. The Python check is told the exact number requested.
## AI on share links
Each share link has two switches, both off by default:
- **Explain this**: visitors can ask for an explanation of a shared question or flashcard.
The request names the item by id; the server loads its text (and checks it belongs to the
share), so a link cannot be used to run arbitrary prompts.
- **Ask about this material**: a chat that answers from the shared item (or, for a shared
class or folder, the item being viewed, the list of materials and passages from its
documents). It cannot read the rest of the library and saves nothing.
Neither has library tools; MCP servers join only when allowed on share links. Both use the
owner's models and keys and show only when an AI model is configured.
AI on a link lasts a week: turning any switch on starts it, and it switches itself off
afterwards (`shares.ai_expires_at`, migration 8). The share dialog shows the end date with
**Extend a week**; once lapsed, **Turn on for a week** brings back the same switches, while
turning a single switch on starts a fresh week with only that feature.
**Region** (Settings → Sharing → AI on public links): Anywhere, United States (default) or
Arizona. Visitors elsewhere still see and study everything, but the page shows no AI and the
API answers 403 `ai_region`. The location comes from Cloudflare's `CF-IPCountry` and
`CF-Region-Code` headers when `TRUST_PROXY` is on, otherwise from the free DB-IP Lite database
(CC BY 4.0), downloaded monthly into `DATA_DIR/geoip` from db-ip.com with the ip-location-db
mirror on jsDelivr as a fallback (country file about 8 MB; Arizona also needs the city file,
over 100 MB, held in memory). Visitor addresses are never sent anywhere. Private and loopback
addresses count as local and are allowed; a public address that can't be placed is refused.
State-level location is approximate: mobile networks can appear to be in another state.
Settings shows the database dates, the last day's allowed and blocked checks, and a warning when
every visitor looks local (the proxy is not passing real addresses).
**Abuse protection** turns a link's AI off (`shares.ai_blocked`, with the reason) when, within
an hour, it is used from 25 different networks or 40 different browsers, gets 60 requests in
five minutes, or keeps being refused (10 requests from outside the region, which the page never
offers, or 30 over the per-device limit). The owner sees why in the share dialog and can turn it
back on. Thresholds are `ABUSE` in `server/routes/shares.ts`.
Limits are per device, not per IP address, so students on one campus network do not share an
allowance: each browser gets a random visitor id in a signed, HTTP-only cookie
(`sd_visitor`, scoped to `/api/public`), limited to 20 requests per 10 minutes per link. A
forged cookie is replaced with a new id. Because clearing cookies gives a fresh allowance, each
link also has a cap of 500 requests per hour across all visitors.
### Generating on share links
Class and exam links have a third visitor switch, **Generate quizzes and flashcards** (off by
default). Visitors choose only the kind, how many (quizzes default to 50, at most 50: one exam;
flashcards default to 30, at most 75), on a class link which exam the set is for (required
when the class has exams; it opens on the latest exam with materials, and the set uses only
that exam's materials and is saved into it), the difficulty, which of the link's class materials to use and,
for quizzes, "Include med math" (5 calculation questions). Quizzes are NCLEX style with 25%
select-all and the Python check. A link may make 4 quizzes and 2 flashcard sets a day (local
midnight, failed attempts not counted), on top of the per-device request limit; **Reset
limits** in the share dialog starts a fresh allowance. Each set is a draft tied to the link
(`drafts.share_id`, migration 7): it shows in the owner's Drafts while it is written, then is
saved into the class (and the exam, for an exam link) as if the owner had approved it, with its
generated name and description. From then on it is ordinary class material: the link lists it
with everything else (marked "New" for two hours), and Explain works on it. While it is written
the link shows it under "In progress" with plain steps ("Verifying questions" rather than the
Python details). A failed set stays there until a visitor dismisses it
(`drafts.share_hidden`); the owner still sees it in Drafts. If the automatic save fails, the
draft waits in Drafts for review. Deleting the link keeps what was made on it (running jobs
finish and save too).
## Original documents
A document's page offers Extracted text, Side by side and Original. PDFs and images show the
file itself. Word and PowerPoint files are shown as a generated page
(`GET /api/documents/:id/preview`): the document's content, or each slide's text, pictures and
notes. Pictures are inlined; formats a browser cannot show (EMF/WMF) are listed as omitted. The
page is served with `default-src 'none'; sandbox`, so it cannot run scripts or load anything.
Pictures inside Word and PowerPoint files are **not read** for search or generation: only
their text, tables and speaker notes are extracted. PDFs are read by OCR only when they have
almost no selectable text, so pictures inside a text PDF are not read either. Material that
lives only in pictures needs to be in the text, or uploaded as an image or scanned PDF and
read with AI.
## ChatGPT account investigation (2026-11-11)
The official [Sign in with ChatGPT quickstart](https://developers.openai.com/siwc/quickstart)
now describes plan access for eligible integrations. The
[official implementation guide](https://developers.openai.com/cookbook/articles/sign-in-with-chatgpt)
distinguishes local personal/open-source tools from remotely hosted apps, which require
access approval. This Unraid/reverse-proxy deployment is a hosted web application; eligibility
for its private client has not been established. Therefore V2 currently uses conventional API
connections, and does not advertise ChatGPT plan access or reuse another client's credentials.
To add this later, obtain the appropriate approval/registration and implement the documented
OAuth/PKCE flow, verified identity, encrypted refresh tokens, approved callbacks and plan-scoped
model discovery. Keep this separate from the single administrator's local login. Never copy
ChatGPT browser session tokens or a Codex client's registration/credentials into Study Desk.
## What has been tested
Mock protocol tests cover request translation, streams, tools, usage and error handling.
Integration tests use a local OpenAI-compatible mock to generate/edit/save drafts, search
documents with citations and exercise confirmed assistant actions, batched OCR of a 20-page
scanned PDF and semantic indexing/search (the mock returns deterministic word-hash vectors).
They cost no API credits.
No successful real-provider inference or real ChatGPT OAuth authorization is claimed.
For manual local AI testing: `npx tsx scripts/mock-ai.ts` starts a scripted mock at
`http://localhost:4010/v1` with key `mock-key`. It returns clearly labeled synthetic material.
Do not use mock output as nursing guidance. Stop the mock and remove its connection after QA.