Study/docs/AI.md
Elijah 25bc03ea75
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 29s
Fourth revision: OpenRouter, live tool stages, Explain layout, share sections
- Interactive blocks are encouraged by default; Explain gets a reduced library
  and never writes practice questions.
- Assistant tool steps breathe while running, show live stages from generation
  and the Python check, and read in the past tense when done.
- The assistant asks before searching the web when class materials do not
  cover a question; clarifying questions list every class's exams and never
  ask about format (always NCLEX).
- Explain opens beside the question from 900px wide, moving the question left
  while open and back on the next one.
- Generate dialog drops the practice set/exam switch (presets cover it) and the
  Generate buttons lose their icon.
- Python check notes are green when every check passed, applied fixes neutral.
- Share pages group by exam, then Quizzes, Flashcards, Notes, Class Materials,
  hiding empty groups; the class page uses the same order. "Documents" is now
  "Class Materials".
- OpenRouter provider replaces the Ollama type; migration 6 turns Ollama
  connections into OpenAI-compatible ones at /v1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 21:54:39 -07:00

16 KiB

AI connections and verification

Settings → AI manages provider connections and model routing. Add a connection, test it, discover models where supported (or add a custom model), enable models and select defaults for generation, chat, explanations and vision. Output limits, capabilities, pricing and reasoning settings can be edited per model. Catalog prices are estimates, not billing records.

Adapters use OpenAI Responses, Anthropic Messages, Gemini generateContent, and Chat Completions for DeepSeek, OpenRouter and OpenAI-compatible servers. Ollama is connected as an OpenAI-compatible server at http://<host>:11434/v1 (migration 6 moved existing Ollama connections there).

OpenRouter (https://openrouter.ai/api/v1) needs a key; its test calls /key, so a bad key fails even though the model list is public. Discovery reads each model's context length, output limit, prices (with cache reads), tool and reasoning support and image input from /models. Reasoning effort is sent as reasoning: { effort }, and requests carry X-Title: Study Desk.

Reasoning effort has seven levels (Off, Minimal, Low, Medium, High, Extra high, Max); each adapter maps them onto the nearest value its API accepts. DeepSeek accepts low / high / max and turns thinking off at "Off". DeepSeek rejects forced tool calls in thinking mode, so structured generation keeps thinking on and asks for the tool call in the prompt; answer keys were noticeably less reliable with thinking off. Its reasoning_content is replayed between tool steps, as the API requires.

Cost estimates split input into fresh, cache-hit and cache-write tokens. A model's "Cached input price" is used when set; otherwise the provider's published discount applies (10% of input for GPT-5+, Claude and Gemini; DeepSeek's catalog cache-hit price). DeepSeek's off-peak half price is applied by time of call. Configure the endpoint as shown in Settings. Local models are reached from the server/container: localhost inside Docker is the container itself, not your workstation or Unraid host. Use a reachable LAN address instead.

Keys are encrypted with AES-256-GCM in SQLite using /data/secret.key or APP_SECRET. Only the server decrypts them. Keep the key with backups; changing it breaks decryption. Documents are sent to the selected provider when you request AI processing; extraction and full-text indexing otherwise run locally. Review generated nursing content and source references before using it for study.

Reading scanned PDFs (OCR)

A PDF with little selectable text is read by the "Vision and OCR" model, which must accept PDFs (OpenAI, Anthropic, Gemini; OpenAI-compatible servers when the model is marked "PDF"). Long files are split with pdf-lib into batches of 8 pages (smaller if a batch exceeds 24 MB) and sent one batch per request, so there is no whole-file size cap. Each batch keeps its real page numbers for citations. Finished batches are cached beside the upload (uploads/<file>.ocr.json), so a restart or a failed batch does not repeat paid work; "Read with AI" on the document forces a fresh read. A batch that fails twice is skipped with a warning on the document. At most 600 pages are read per file.

Settings → AI → Semantic search chooses an embeddings provider and model (suggested: OpenAI text-embedding-3-small, Gemini gemini-embedding-001, or e.g. nomic-embed-text on Ollama through an OpenAI-compatible connection; Anthropic, DeepSeek and OpenRouter connections are not used for embeddings). Until something else is chosen, the built-in model is used: BAAI bge-small-en-v1.5 (8-bit ONNX, 384 dimensions) running on the server's CPU through transformers.js. The Docker image ships the model in /app/models (LOCAL_MODELS_DIR), so it works without internet access; outside Docker it is downloaded once (about 35 MB) from huggingface.co into DATA_DIR/models. It costs nothing. If the model cannot load, Settings shows the actual error. LOCAL_EMBEDDINGS=off disables the default. Every document passage is embedded in the background and stored as a vector in SQLite (chunk_vectors). Search, the assistant's document search and focused generation then merge keyword (FTS5) and meaning-based rankings with reciprocal-rank fusion, so exact terms still match while paraphrased questions find the right passage. If the embedding provider fails, search silently falls back to keywords. Changing the model clears and rebuilds the index. Costs appear under Usage as "semantic indexing" / "semantic search". Quiz questions, flashcards and notes are still searched by keyword.

Python sandbox and the quiz check

The assistant has a run_python tool, and generated quizzes are checked with it. Code runs in Pyodide (CPython compiled to WebAssembly) inside a separate Node process started with --permission. It can read only the Pyodide files, gets an empty environment, cannot reach the network or start processes (blocked in Python and again in JavaScript), and has no access to /data. One warm process is reused and recycled every 50 runs; a run that times out kills it.

After a quiz is generated (unless "Check answers" is off), the generation model gets the draft as Python data plus the request (count, select-all share, calculation allowance). In at most three Python runs it checks the count, question types, number of correct answers per type, numeric answers, the calculation allowance and topic breadth, then reports with submit_review. Fixes are applied only when the question stays valid. Remarks the model files as cosmetic or informational (answer position, quotes) are not counted as failed checks; answer order is shuffled whenever a quiz is taken.

Quiz generation rules

Every quiz is NCLEX style. "Practice exam" writes 50 questions. Calculation (numeric) questions are written only when the source material involves calculations (dosage, drip rates, mg/kg, intake and output...): up to 5 in an exam, about one in ten otherwise, none when the material has no calculations. Topics are kept broad ("Stroke", not "Right hemisphere stroke"): the prompt lists the class's existing topics, and narrower labels that contain an existing one are merged into it after generation.

MCP servers

Settings → AI → MCP servers connects Model Context Protocol servers over Streamable HTTP (JSON-RPC over POST, answered as JSON or SSE). Headers (usually an Authorization token) are encrypted like API keys and never returned to the browser. "Test and list tools" connects afresh and lists the tools.

Each server has:

  • When to use it: your instructions, added to the model's prompt next to the server's tools.
  • Where it can be used: Assistant chat, Explain this, Generating material (quizzes, flashcards, study guides), Quiz check and Share links, each switched separately. New servers are on for chat and Explain only. "Share links" lets visitors of links with AI turned on use the server through your account; leave it off for servers with private data or paid keys.

Tools are offered as mcp_<server>__<tool>, at most 48 per request. A server that cannot be reached is skipped for that request, and the assistant is told it is unavailable. During generation the model may make up to four rounds of tool calls before it must submit its answer; the quiz check allows four external calls. Tool output is treated as data, not instructions. On share links only servers with "Share links" switched on are offered.

Signing in to MCP servers (OAuth)

Any server that answers 401 with a WWW-Authenticate: Bearer resource_metadata=… challenge shows Sign in in its row. The app follows the MCP authorization spec (2025-06-18):

  1. Reads the protected-resource metadata (RFC 9728), then the authorization server's metadata (RFC 8414, falling back to OpenID discovery).
  2. Registers itself by dynamic client registration (RFC 7591), unless a client ID (and optional secret) is entered under the server's advanced settings. Registrations are kept and reused.
  3. Sends the browser to the sign-in page with PKCE S256, state and the resource indicator (RFC 8707). The callback (GET /api/ai/mcp/oauth/callback) exchanges the code and returns to the page that started it.

Tokens are encrypted like API keys, refreshed when within a minute of expiry, and retried once after a 401. Sign out forgets the tokens. Pending sign-ins expire after 15 minutes.

Settings → Web search is a built-in entry backed by Parallel's MCP server (https://mcp.parallel.ai/mcp, override with WEB_SEARCH_URL). It works without an account on Parallel's free tier; Sign in with Parallel uses the same OAuth flow against /mcp?login for higher limits, and then offers deep research (web_research) when switched on. Only web_search and web_fetch (plus research) are offered; Parallel's bookkeeping arguments (session_id, client_model) are hidden from the model, and ids, usage and warnings are stripped from results. It is on for chat and Explain by default (off in tests), with its own instructions and area switches like any server.

Tool steps

Every tool call shows as one line: an icon for the kind of tool, a friendly name ("Searched the web for: …", "Read pubmed.ncbi.nlm.nih.gov", "Drug reference: Lookup drug") and a check or cross. While a step runs its icon breathes, and long steps report their stage (generation and Python checking send "Reading your material", "Writing questions 1-8 of 15", "Checking the questions with Python" and so on through ToolActivity.stage). Finished steps read in the past tense ("Wrote 15 quiz questions"). Clicking it shows the pages found, the request and a clipped result. More than three finished steps fold into "N steps". The model is told never to mention tool names or internal ids. In a chat, the sliders button left of + turns individual servers (and web search) off for that conversation; the choice is saved with it.

Clarifying questions

The assistant's ask_user tool shows up to four questions as an inline form (choices, several choices, text or a number, with defaults) and ends the turn. The answers are sent as the user's next message, and the stored tool result travels with that message, so the model sees question and answer together. The prompt tells the model to ask before generating when the exam, materials or count are missing and not clear from the page in view, to ask at most once per request, and to use defaults when the user says to go ahead. Every class is listed in the prompt with its exams, so the exam question can offer them by name; the model is told never to ask about question format (quizzes are always NCLEX style).

When web search is on in a chat and the class materials do not answer a question, the assistant asks whether to search the web ("Search the web" / "Answer from general knowledge") instead of answering from its own knowledge. It searches straight away only when asked to.

Exams within a class

A class is divided into exams (or units, modules, weeks, topics or folders: Class settings → "Divided into"). Exams are the existing folders; classes.group_label holds the name. The class page shows one tab per exam plus "Not in an exam" and "All", opening on the last exam (the choice is remembered per class). Uploads, pasted text, imports, new items and Generate default to the exam in view. Generating for an exam uses only that exam's documents (also for weak areas and searches), and the draft is saved in that exam; an exam from another class is refused. The assistant is told the class's exams with their ids, which exam is in view, and passes exam_id to its generate and search tools.

Intelligent UI in chats

Assistant replies (owner, Explain and share-link chats) can include interactive blocks written in OpenUI Lang inside ```openui-lang fences, rendered with @openuidev/react-lang against the app's own component library (shared/genui.ts lists it; web/src/genui/ renders it): layout, callouts, stats, tables, steps, bar/line/pie charts with a table view, sliders and inputs bound to variables, practice questions, flashcards and follow-up buttons that send a message. The prompt lists the reply shapes that should become a block (comparisons, sequences, calculations, safety points, a self-check after teaching), so blocks appear without being asked for. Explain uses a reduced library without Question and Flashcards and is told to give information only (genUiPrompt('explain')). Library telemetry is off (OPENUI_TELEMETRY_DISABLED=1, DO_NOT_TRACK=1).

Each share link has two switches, both off by default:

  • Explain this: visitors can ask for an explanation of a shared question or flashcard. The request names the item by id; the server loads its text (and checks it belongs to the share), so a link cannot be used to run arbitrary prompts.
  • Ask about this material: a chat that answers from the shared item (or, for a shared class or folder, the item being viewed, the list of materials and passages from its documents). It cannot read the rest of the library and saves nothing.

Neither has library tools; MCP servers join only when allowed on share links. Both use the owner's models and keys and show only when an AI model is configured.

Limits are per device, not per IP address, so students on one campus network do not share an allowance: each browser gets a random visitor id in a signed, HTTP-only cookie (sd_visitor, scoped to /api/public), limited to 20 requests per 10 minutes per link. A forged cookie is replaced with a new id. Because clearing cookies gives a fresh allowance, each link also has a cap of 500 requests per hour across all visitors.

Original documents

A document's page offers Extracted text, Side by side and Original. PDFs and images show the file itself. Word and PowerPoint files are shown as a generated page (GET /api/documents/:id/preview): the document's content, or each slide's text, pictures and notes. Pictures are inlined; formats a browser cannot show (EMF/WMF) are listed as omitted. The page is served with default-src 'none'; sandbox, so it cannot run scripts or load anything.

Pictures inside Word and PowerPoint files are not read for search or generation: only their text, tables and speaker notes are extracted. PDFs are read by OCR only when they have almost no selectable text, so pictures inside a text PDF are not read either. Material that lives only in pictures needs to be in the text, or uploaded as an image or scanned PDF and read with AI.

ChatGPT account investigation (2026-11-11)

The official Sign in with ChatGPT quickstart now describes plan access for eligible integrations. The official implementation guide distinguishes local personal/open-source tools from remotely hosted apps, which require access approval. This Unraid/reverse-proxy deployment is a hosted web application; eligibility for its private client has not been established. Therefore V2 currently uses conventional API connections, and does not advertise ChatGPT plan access or reuse another client's credentials.

To add this later, obtain the appropriate approval/registration and implement the documented OAuth/PKCE flow, verified identity, encrypted refresh tokens, approved callbacks and plan-scoped model discovery. Keep this separate from the single administrator's local login. Never copy ChatGPT browser session tokens or a Codex client's registration/credentials into Study Desk.

What has been tested

Mock protocol tests cover request translation, streams, tools, usage and error handling. Integration tests use a local OpenAI-compatible mock to generate/edit/save drafts, search documents with citations and exercise confirmed assistant actions, batched OCR of a 20-page scanned PDF and semantic indexing/search (the mock returns deterministic word-hash vectors). They cost no API credits. No successful real-provider inference or real ChatGPT OAuth authorization is claimed.

For manual local AI testing: npx tsx scripts/mock-ai.ts starts a scripted mock at http://localhost:4010/v1 with key mock-key. It returns clearly labeled synthetic material. Do not use mock output as nursing guidance. Stop the mock and remove its connection after QA.