diff --git a/PROGRESS.md b/PROGRESS.md index 0c5d137..406d24c 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -204,8 +204,9 @@ sheet on phones. The drawer is resizable for both (drag the edge, 340-760 px, do reset, remembered per browser). Checked in the browser at 1440, 1100 and 375 px wide, with a live answer streamed on a share link. -Settings → Data → Import now restores a full backup zip in place of the V1 import row (the -`/api/import/v1` endpoint stays for the V1 cutover; see docs/MIGRATION.md). The restore replaces +Settings → Data → Import now restores a full backup zip in place of the V1 import row. The V1 +database importer was then removed entirely (route, service and its test; it remains in git +history at `7125dec`, see docs/MIGRATION.md). The restore replaces the database, settings and files without a restart, re-encrypts saved keys for this server, keeps a `pre-restore-*` snapshot and keeps the person signed in. Tests: `tests/restore.test.ts`; round trip on the rev1 copy: 1 MB backup restored in 75 ms, same library, still signed in. @@ -218,7 +219,7 @@ stored DeepSeek connection still holding the old key; it was updated from `.env` ## Verification status Baseline: 162 tests passed, TypeScript passed and production web/server builds passed. -Added six regression tests. Full `npm run check` passes: **291 tests** (full-backup restore, 2026-11-16; 289 after rationale layout; 285 after varied openers; 281 after generation steps; 280 after the check loop and second review; 274 after the fifth revision; 261 after share-link generation, 2026-11-15; 257 after the fourth revision; 247 after the third, 208 after the second, 181 on 2026-11-12), TypeScript +Added six regression tests. Full `npm run check` passes: **290 tests** (V1 importer removed, 2026-11-16; 291 after full-backup restore; 289 after rationale layout; 285 after varied openers; 281 after generation steps; 280 after the check loop and second review; 274 after the fifth revision; 261 after share-link generation, 2026-11-15; 257 after the fourth revision; 247 after the third, 208 after the second, 181 on 2026-11-12), TypeScript and both production builds. The Docker `test` stage runs the same 181 tests on Node 24. Tests were run with loopback access for the mock provider. The earlier agent manually exercised quiz completion/results, flashcards, streak effects, diff --git a/README.md b/README.md index a77d666..d319079 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ is an opt-in development tool, never part of startup. allow "Explain this" and a chat about the shared material, separately (off by default). - Flashcards in Standard mode (know it / still learning, optionally repeating missed cards until known) or spaced repetition. -- JSON import previews, validation, templates, content exports and a V1 SQLite import path. +- JSON import previews, validation, templates, content exports and full-backup restore from Settings. - PDF/DOCX/PPTX/TXT/Markdown/image uploads; text extraction, OCR via a configured vision model, chunking and SQLite FTS retrieval with source references. The original is shown next to the extracted text (Word and PowerPoint as a generated preview). @@ -73,7 +73,7 @@ is an opt-in development tool, never part of startup. ## Deploy, back up, migrate -See [deployment and restore](docs/DEPLOYMENT.md), [V1 migration](docs/MIGRATION.md), and +See [deployment and restore](docs/DEPLOYMENT.md), [moving data into V2](docs/MIGRATION.md), and [AI providers and account integration](docs/AI.md). The image runs a single Node process with SQLite and local file storage. No database or queue diff --git a/docs/MIGRATION.md b/docs/MIGRATION.md index 928a36e..de616c8 100644 --- a/docs/MIGRATION.md +++ b/docs/MIGRATION.md @@ -1,25 +1,27 @@ -# Importing V1 safely +# Moving data into V2 -V1 code remains on `main` at `7bd0a9a` and in history. V2 has a separate schema and must use -a separate data folder. It never upgrades a production V1 database in place. +V2 has its own schema and must use its own data folder. It never opens or upgrades a V1 +database. -1. Use V1's backup facility to make a consistent SQLite copy. If copying files yourself, - stop V1 first; copying only a live WAL-mode `.db` can omit recent writes. -2. Keep the original and a verified backup untouched. Start V2 with an empty V2 volume. -3. Settings no longer shows the V1 import (its place went to restoring a full V2 backup), but - the endpoint remains. While signed in, preview the **backup copy** with - `POST /api/import/v1?preview=1` (multipart field `file`, header `X-Requested-With: studydesk`), - then import it without `?preview=1`. -4. Review the reported counts/skipped items, then explicitly import. V2 opens the submitted - SQLite file read-only and creates new IDs in one transaction. A failed import rolls back. -5. Compare class/folder/deck/quiz counts and representative questions, rationales, attempts and - scheduling. V1's passwords, provider settings and share URLs are not reused. Recreate links - deliberately. Running the import again creates another set of content. +## From V1 -The importer supports V1 classes, material groups, decks/cards, quiz sets/questions/options, -completed attempts, spaced-repetition card state and activity where present. It reports -unsupported/skipped records. Older optional game features are not migrated as games. -V1 JSON quiz/flashcard exports can also be pasted/uploaded in a class's Import dialog. +V2 no longer imports V1 databases. The importer (`server/services/v1import.ts` and the +`/api/import/v1` endpoint) was removed on 2026-11-16, and Settings → Data now restores full V2 +backups instead. Two ways remain for V1 content: -Portable V2 JSON exports are useful for migration between apps; full ZIP backups are needed -to preserve progress, documents, settings and credentials. See DEPLOYMENT.md for recovery. +- **V1 JSON exports.** V1 quiz and flashcard exports can be pasted or uploaded in a class's + Import dialog. This brings content only: no attempts, schedules or activity. +- **The old importer.** It is in git history, last as of commit `7125dec`. Check that commit out + into a separate working copy and run it against an empty V2 data folder. Point it at a + consistent **backup copy** of the V1 database, never the live file. Make a full V2 backup + afterwards, then restore that backup on the current version. + +V1 code remains on `main` at `7bd0a9a` and in history. + +## Between V2 installs + +- **Full backup ZIP** (Settings → Data → Download backup, then Restore a full backup on the + other server): everything, including progress, documents, settings and AI credentials. See + DEPLOYMENT.md. +- **Portable JSON export**: classes, quizzes, flashcards and notes only, added alongside what + the other install already has. diff --git a/server/routes/data.ts b/server/routes/data.ts index a6322d0..f7b4447 100644 --- a/server/routes/data.ts +++ b/server/routes/data.ts @@ -1,5 +1,4 @@ import fs from 'node:fs'; -import os from 'node:os'; import path from 'node:path'; import { Hono } from 'hono'; import { bodyLimit } from 'hono/body-limit'; @@ -12,7 +11,6 @@ import { createSnapshot, exportBundle, fullBackupStream, listBackups } from '../ import { createClass, createDeck, createFolder, createNote, createQuiz, draftCardToInput, draftQuestionToInput, getClass, getDeck, getQuiz, listCards, listQuestions } from '../services/content.ts'; import { createTemplate, deleteTemplate, listTemplates, resetTemplate, templateInputSchema, updateTemplate } from '../services/templates.ts'; import { restoreFullBackup } from '../services/restore.ts'; -import { importV1 } from '../services/v1import.ts'; import { sessionHash } from '../auth.ts'; import { slugify } from '../util.ts'; @@ -215,29 +213,6 @@ export function dataRoutes(app: AppContext): Hono { } }); - /** - * Migration from a V1 database file. `?preview=1` only reports what would be imported. Not in - * Settings any more (full-backup restore took its place); see docs/MIGRATION.md. - */ - r.post('/import/v1', bodyLimit({ maxSize: 500 * 1024 * 1024 }), async (c) => { - const form = await c.req.parseBody().catch(() => { - throw badRequest('Upload could not be read'); - }); - const file = form.file; - if (!file || typeof file === 'string') throw badRequest('Choose your V1 study.db file'); - const bytes = new Uint8Array(await file.arrayBuffer()); - if (new TextDecoder().decode(bytes.subarray(0, 15)) !== 'SQLite format 3') throw badRequest('That file is not a SQLite database.'); - const tmp = path.join(os.tmpdir(), `studydesk-v1-${Date.now()}-${Math.random().toString(36).slice(2)}.db`); - fs.writeFileSync(tmp, bytes); - try { - const preview = c.req.query('preview') === '1'; - if (!preview) createSnapshot(app, 'pre-v1-import'); - return c.json({ preview, summary: importV1(app, tmp, { dryRun: preview }) }); - } finally { - fs.rmSync(tmp, { force: true }); - } - }); - /* ------------------------------ export + backup ------------------------------ */ r.get('/export/all', (c) => { diff --git a/server/services/v1import.ts b/server/services/v1import.ts deleted file mode 100644 index 800e9e2..0000000 --- a/server/services/v1import.ts +++ /dev/null @@ -1,258 +0,0 @@ -/** - * One-way migration from a Study Desk V1 database (the Prisma/SQLite `study.db`). - * The V1 file is opened read-only and never modified. Everything is copied into new - * rows with new ids inside one transaction, so a failed import leaves V2 untouched. - */ -import { DatabaseSync } from 'node:sqlite'; -import { isFullyCorrect, scoreQuestion, type Option } from '../../shared/content.ts'; -import { badRequest, type AppContext } from '../http.ts'; -import { newId, now } from '../util.ts'; -import { createClass, createDeck, createFolder, createQuiz, listCards, listQuestions } from './content.ts'; - -export interface V1Summary { - classes: number; - folders: number; - decks: number; - cards: number; - quizzes: number; - questions: number; - attempts: number; - scheduledCards: number; - activityDays: number; - skipped: string[]; -} - -type Row = Record; - -function toMs(v: unknown): number | null { - if (v === null || v === undefined) return null; - if (typeof v === 'number') return v > 1e11 ? v : v * 1000; - if (typeof v === 'bigint') return Number(v); - const n = Number(v); - if (Number.isFinite(n) && String(v).trim() !== '') return n > 1e11 ? n : n * 1000; - const t = Date.parse(String(v)); - return Number.isNaN(t) ? null : t; -} - -export function importV1(app: AppContext, file: string, opts: { dryRun?: boolean } = {}): V1Summary { - let src: DatabaseSync; - try { - src = new DatabaseSync(file, { readOnly: true }); - } catch { - throw badRequest('That file could not be opened as a SQLite database.'); - } - try { - const tables = new Set((src.prepare(`SELECT name FROM sqlite_master WHERE type = 'table'`).all() as Row[]).map((r) => String(r.name))); - if (!tables.has('Class') || !tables.has('Deck') || !tables.has('QuizSet')) { - throw badRequest('This does not look like a Study Desk V1 database (expected the tables Class, Deck and QuizSet).'); - } - const all = (table: string, order = ''): Row[] => (tables.has(table) ? (src.prepare(`SELECT * FROM "${table}" ${order}`).all() as Row[]) : []); - const classes = all('Class', 'ORDER BY sortOrder, createdAt'); - const groups = all('MaterialGroup', 'ORDER BY sortOrder, createdAt'); - const decks = all('Deck', 'ORDER BY sortOrder, createdAt'); - const cards = all('Flashcard', 'ORDER BY sortOrder'); - const quizzes = all('QuizSet', 'ORDER BY sortOrder, createdAt'); - const questions = all('Question', 'ORDER BY sortOrder'); - const options = all('AnswerOption', 'ORDER BY sortOrder'); - const attempts = all('QuizAttempt', 'ORDER BY completedAt'); - const srStates = all('SpacedRepetitionCardState'); - const activity = all('StudyActivity'); - - const summary: V1Summary = { - classes: classes.length, - folders: groups.length, - decks: decks.length, - cards: cards.length, - quizzes: quizzes.length, - questions: questions.length, - attempts: attempts.length, - scheduledCards: new Set(srStates.map((s) => s.flashcardId)).size, - activityDays: new Set(activity.map((a) => new Date(toMs(a.occurredAt) ?? 0).toISOString().slice(0, 10))).size, - skipped: [], - }; - if (opts.dryRun) return summary; - - const { db } = app; - const group = (rows: T[], key: string) => { - const m = new Map(); - for (const r of rows) { - const k = String(r[key]); - if (!m.has(k)) m.set(k, []); - m.get(k)!.push(r); - } - return m; - }; - const cardsByDeck = group(cards, 'deckId'); - const questionsByQuiz = group(questions, 'quizSetId'); - const optionsByQuestion = group(options, 'questionId'); - const attemptsByQuiz = group(attempts, 'quizSetId'); - - // Keep the most-practised schedule when a card belonged to several V1 spaced-repetition sets. - const bestState = new Map(); - for (const s of srStates) { - const cur = bestState.get(s.flashcardId); - if (!cur || Number(s.reps) > Number(cur.reps)) bestState.set(s.flashcardId, s); - } - - db.tx(() => { - const classMap = new Map(); - const folderMap = new Map(); - for (const c of classes) { - const created = createClass(db, { name: String(c.name || 'Untitled class').slice(0, 120) }); - classMap.set(c.id, created.id); - const t = toMs(c.createdAt); - if (t) db.run('UPDATE classes SET created_at = ? WHERE id = ?', t, created.id); - } - for (const g of groups) { - const classId = classMap.get(g.classId); - if (!classId) continue; - folderMap.set(g.id, createFolder(db, classId, String(g.name || 'Folder')).id); - } - - for (const d of decks) { - const classId = classMap.get(d.classId); - if (!classId) { - summary.skipped.push(`Deck "${d.name}" (its class is missing)`); - continue; - } - const source = (cardsByDeck.get(d.id) ?? []).filter((c) => String(c.front ?? '').trim() && String(c.back ?? '').trim()); - const deck = createDeck(db, { - classId, - folderId: d.groupId ? folderMap.get(d.groupId) : null, - title: String(d.name || 'Untitled deck').slice(0, 200), - description: String(d.description ?? '').slice(0, 4000), - cards: source.map((c) => ({ front: String(c.front).slice(0, 20000), back: String(c.back).slice(0, 20000) })), - }); - const created = listCards(db, deck.id); - source.forEach((old, i) => { - const s = bestState.get(old.id); - const target = created[i]; - if (!s || !target) return; - db.run( - `UPDATE cards SET state = ?, due = ?, stability = ?, difficulty = ?, elapsed_days = ?, scheduled_days = ?, learning_steps = ?, reps = ?, lapses = ?, last_review = ?, introduced_at = ? WHERE id = ?`, - Number(s.state) || 0, - toMs(s.due), - Number(s.stability) || 0, - Number(s.difficulty) || 0, - Number(s.elapsedDays) || 0, - Number(s.scheduledDays) || 0, - Number(s.learningSteps) || 0, - Number(s.reps) || 0, - Number(s.lapses) || 0, - toMs(s.lastReview), - toMs(s.firstReviewedAt), - target.id, - ); - }); - } - - for (const z of quizzes) { - const classId = classMap.get(z.classId); - if (!classId) { - summary.skipped.push(`Quiz "${z.name}" (its class is missing)`); - continue; - } - const source = (questionsByQuiz.get(z.id) ?? []).filter((q) => { - const opts2 = optionsByQuestion.get(q.id) ?? []; - return String(q.prompt ?? '').trim() && opts2.length >= 2 && opts2.some((o) => o.isCorrect); - }); - const quiz = createQuiz(db, { - classId, - folderId: z.groupId ? folderMap.get(z.groupId) : null, - title: String(z.name || 'Untitled quiz').slice(0, 200), - description: String(z.description ?? '').slice(0, 4000), - questions: source.map((q) => { - const opts2 = (optionsByQuestion.get(q.id) ?? []).filter((o) => String(o.text ?? '').trim()).slice(0, 10); - const correct = opts2.filter((o) => o.isCorrect).length; - return { - type: String(q.type).toUpperCase() === 'SATA' || correct > 1 ? 'multi' : 'single', - stem: String(q.prompt).slice(0, 20000), - options: opts2.map((o) => ({ text: String(o.text).slice(0, 4000), correct: Boolean(o.isCorrect) })), - rationale: String(q.rationale ?? '').slice(0, 20000), - topic: String(q.category ?? '').slice(0, 80), - }; - }), - }); - const created = listQuestions(db, quiz.id); - // Old question / option ids -> new, so past attempts can be replayed. - const qMap = new Map(); - const oMap = new Map(); - source.forEach((old, i) => { - const target = created[i]; - if (!target) return; - qMap.set(old.id, target); - const oldOpts = (optionsByQuestion.get(old.id) ?? []).filter((o) => String(o.text ?? '').trim()).slice(0, 10); - oldOpts.forEach((o, j) => target.options[j] && oMap.set(o.id, target.options[j].id)); - }); - - for (const a of attemptsByQuiz.get(z.id) ?? []) { - let answers: Record = {}; - try { - const parsed = JSON.parse(String(a.answersJson ?? '{}')); - if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) answers = parsed; - } catch { - /* an unreadable attempt is still imported as a score */ - } - const partial = Boolean(a.isPartialRetake); - const included = created.filter((q) => { - const oldId = [...qMap.entries()].find(([, v]) => v.id === q.id)?.[0]; - return !partial || (oldId !== undefined && oldId in answers); - }); - const completedAt = toMs(a.completedAt) ?? now(); - const newAnswers: Record = {}; - for (const [oldQ, sel] of Object.entries(answers)) { - const target = qMap.get(oldQ); - if (!target || !Array.isArray(sel)) continue; - newAnswers[target.id] = { selected: sel.map((s) => oMap.get(String(s))).filter((s): s is string => Boolean(s)), checked: true, at: completedAt }; - } - const attemptId = newId(); - const snapshot = included.map((q) => ({ id: q.id, quizId: quiz.id, quizTitle: quiz.title, type: q.type, stem: q.stem, options: q.options as Option[], rationale: q.rationale, topic: q.topic, difficulty: q.difficulty })); - db.run( - `INSERT INTO attempts (id, quiz_id, class_id, title, mode, kind, status, settings, snapshot, answers, question_count, score, max_score, started_at, updated_at, completed_at) - VALUES (?, ?, ?, ?, 'practice', ?, 'completed', ?, ?, ?, ?, ?, ?, ?, ?, ?)`, - attemptId, - quiz.id, - classId, - quiz.title, - partial ? 'retake' : 'full', - JSON.stringify({ shuffleQuestions: false, shuffleOptions: false, timeLimitSec: null, sataScoring: 'partial' }), - JSON.stringify({ questions: snapshot }), - JSON.stringify(newAnswers), - snapshot.length, - Number(a.score) || 0, - Number(a.maxScore) || snapshot.length, - completedAt, - completedAt, - completedAt, - ); - for (const q of snapshot) { - const selected = newAnswers[q.id]?.selected ?? []; - if (selected.length === 0 && partial) continue; - const score = scoreQuestion(q, selected, 'partial'); - db.run( - 'INSERT OR IGNORE INTO responses (attempt_id, question_id, quiz_id, class_id, topic, score, correct, answered_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)', - attemptId, - q.id, - quiz.id, - classId, - q.topic, - score, - isFullyCorrect(score) ? 1 : 0, - completedAt, - ); - } - } - } - - // Flashcard activity keeps the study heatmap and streak intact. Quiz activity is rebuilt from the attempts above. - for (const a of activity) { - if (String(a.type) !== 'FLASHCARD') continue; - const t = toMs(a.occurredAt); - if (t) db.run(`INSERT INTO reviews (card_id, deck_id, class_id, rating, state_before, scheduled_days, mode, prev, reviewed_at) VALUES ('v1', NULL, NULL, 3, 0, 0, 'legacy', '', ?)`, t); - } - }); - return summary; - } finally { - src.close(); - } -} diff --git a/tests/migration.test.ts b/tests/migration.test.ts index 696cc5b..66310ee 100644 --- a/tests/migration.test.ts +++ b/tests/migration.test.ts @@ -1,61 +1,11 @@ -import crypto from 'node:crypto'; import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; -import { DatabaseSync } from 'node:sqlite'; import { describe, expect, it } from 'vitest'; import { DB, migrate } from '../server/db.ts'; import { MIGRATIONS } from '../server/migrations.ts'; -import { importV1 } from '../server/services/v1import.ts'; -import { createTestApp } from './helpers.ts'; describe('safe migration paths', () => { - it('previews and imports a V1 database without changing its bytes, preserving answers and schedules', async () => { - const app = await createTestApp(); - const file = path.join(app.dir, 'v1-fixture.db'); - const old = new DatabaseSync(file); - old.exec(` - CREATE TABLE Class (id TEXT, name TEXT, sortOrder INTEGER, createdAt INTEGER); - CREATE TABLE MaterialGroup (id TEXT, classId TEXT, name TEXT, sortOrder INTEGER, createdAt INTEGER); - CREATE TABLE Deck (id TEXT, classId TEXT, groupId TEXT, name TEXT, description TEXT, sortOrder INTEGER, createdAt INTEGER); - CREATE TABLE Flashcard (id TEXT, deckId TEXT, front TEXT, back TEXT, sortOrder INTEGER); - CREATE TABLE QuizSet (id TEXT, classId TEXT, groupId TEXT, name TEXT, description TEXT, sortOrder INTEGER, createdAt INTEGER); - CREATE TABLE Question (id TEXT, quizSetId TEXT, type TEXT, prompt TEXT, rationale TEXT, category TEXT, sortOrder INTEGER); - CREATE TABLE AnswerOption (id TEXT, questionId TEXT, text TEXT, isCorrect INTEGER, sortOrder INTEGER); - CREATE TABLE QuizAttempt (id TEXT, quizSetId TEXT, score REAL, maxScore INTEGER, answersJson TEXT, isPartialRetake INTEGER, completedAt INTEGER); - CREATE TABLE SpacedRepetitionCardState (flashcardId TEXT, state INTEGER, reps INTEGER, due INTEGER, stability REAL, difficulty REAL, scheduledDays INTEGER, learningSteps INTEGER, lapses INTEGER, lastReview INTEGER, firstReviewedAt INTEGER, elapsedDays INTEGER); - INSERT INTO Class VALUES ('c','Legacy class',0,1700000000000); - INSERT INTO MaterialGroup VALUES ('g','c','Legacy group',0,1700000000000); - INSERT INTO Deck VALUES ('d','c','g','Legacy deck','',0,1700000000000); - INSERT INTO Flashcard VALUES ('f','d','Term','Definition',0); - INSERT INTO QuizSet VALUES ('z','c','g','Legacy quiz','',0,1700000000000); - INSERT INTO Question VALUES ('q','z','MULTIPLE_CHOICE','Pick A','Reason','Topic',0); - INSERT INTO AnswerOption VALUES ('a','q','A',1,0),('b','q','B',0,1); - INSERT INTO QuizAttempt VALUES ('attempt','z',1,1,'{"q":["a"]}',0,1700000000000); - INSERT INTO SpacedRepetitionCardState VALUES ('f',2,7,1800000000000,15,5,14,0,1,1700000000000,1600000000000,12); - `); - old.close(); - const digest = () => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex'); - const before = digest(); - try { - expect(importV1(app.ctx, file, { dryRun: true })).toMatchObject({ classes: 1, quizzes: 1, cards: 1, attempts: 1 }); - expect(app.ctx.db.get<{ n: number }>('SELECT count(*) AS n FROM classes')?.n).toBe(0); - importV1(app.ctx, file); - expect(digest()).toBe(before); - const classes = (await app.api('GET', '/api/classes')).body.classes; - expect(classes[0].name).toBe('Legacy class'); - const card = app.ctx.db.get<{ reps: number; due: number; elapsed_days: number }>('SELECT reps, due, elapsed_days FROM cards'); - expect(card).toMatchObject({ reps: 7, due: 1800000000000, elapsed_days: 12 }); - const attempt = app.ctx.db.get<{ id: string }>('SELECT id FROM attempts')!; - const result = (await app.api('GET', `/api/attempts/${attempt.id}`)).body; - expect(JSON.stringify(result)).toContain('Pick A'); - expect(app.ctx.db.get<{ score: number }>('SELECT score FROM responses')?.score).toBe(1); - expect(app.ctx.db.all('PRAGMA foreign_key_check')).toEqual([]); - } finally { - app.close(); - } - }); - it('backs up before a schema change and rolls back a failed migration', () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'studydesk-migration-test-')); const db = new DB(path.join(dir, 'test.db'));