Remove the V1 database importer
All checks were successful
Verify and publish V2 container / build-and-push (push) Successful in 30s

The /api/import/v1 endpoint, server/services/v1import.ts and its test are
gone; Settings already restores full V2 backups in that place. MIGRATION.md
now covers V1 JSON exports and running the old importer from history
(7125dec), plus moving data between V2 installs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Elijah 2026-11-16 12:35:51 -07:00
parent 7125dec3e0
commit 5fef59b07b
6 changed files with 29 additions and 359 deletions

View file

@ -204,8 +204,9 @@ sheet on phones. The drawer is resizable for both (drag the edge, 340-760 px, do
reset, remembered per browser). Checked in the browser at 1440, 1100 and 375 px wide, with a live
answer streamed on a share link.
Settings → Data → Import now restores a full backup zip in place of the V1 import row (the
`/api/import/v1` endpoint stays for the V1 cutover; see docs/MIGRATION.md). The restore replaces
Settings → Data → Import now restores a full backup zip in place of the V1 import row. The V1
database importer was then removed entirely (route, service and its test; it remains in git
history at `7125dec`, see docs/MIGRATION.md). The restore replaces
the database, settings and files without a restart, re-encrypts saved keys for this server, keeps
a `pre-restore-*` snapshot and keeps the person signed in. Tests: `tests/restore.test.ts`;
round trip on the rev1 copy: 1 MB backup restored in 75 ms, same library, still signed in.
@ -218,7 +219,7 @@ stored DeepSeek connection still holding the old key; it was updated from `.env`
## Verification status
Baseline: 162 tests passed, TypeScript passed and production web/server builds passed.
Added six regression tests. Full `npm run check` passes: **291 tests** (full-backup restore, 2026-11-16; 289 after rationale layout; 285 after varied openers; 281 after generation steps; 280 after the check loop and second review; 274 after the fifth revision; 261 after share-link generation, 2026-11-15; 257 after the fourth revision; 247 after the third, 208 after the second, 181 on 2026-11-12), TypeScript
Added six regression tests. Full `npm run check` passes: **290 tests** (V1 importer removed, 2026-11-16; 291 after full-backup restore; 289 after rationale layout; 285 after varied openers; 281 after generation steps; 280 after the check loop and second review; 274 after the fifth revision; 261 after share-link generation, 2026-11-15; 257 after the fourth revision; 247 after the third, 208 after the second, 181 on 2026-11-12), TypeScript
and both production builds. The Docker `test` stage runs the same 181 tests on Node 24. Tests were run with loopback access for the mock provider.
The earlier agent manually exercised quiz completion/results, flashcards, streak effects,

View file

@ -45,7 +45,7 @@ is an opt-in development tool, never part of startup.
allow "Explain this" and a chat about the shared material, separately (off by default).
- Flashcards in Standard mode (know it / still learning, optionally repeating missed cards
until known) or spaced repetition.
- JSON import previews, validation, templates, content exports and a V1 SQLite import path.
- JSON import previews, validation, templates, content exports and full-backup restore from Settings.
- PDF/DOCX/PPTX/TXT/Markdown/image uploads; text extraction, OCR via a configured vision model,
chunking and SQLite FTS retrieval with source references. The original is shown next to the
extracted text (Word and PowerPoint as a generated preview).
@ -73,7 +73,7 @@ is an opt-in development tool, never part of startup.
## Deploy, back up, migrate
See [deployment and restore](docs/DEPLOYMENT.md), [V1 migration](docs/MIGRATION.md), and
See [deployment and restore](docs/DEPLOYMENT.md), [moving data into V2](docs/MIGRATION.md), and
[AI providers and account integration](docs/AI.md).
The image runs a single Node process with SQLite and local file storage. No database or queue

View file

@ -1,25 +1,27 @@
# Importing V1 safely
# Moving data into V2
V1 code remains on `main` at `7bd0a9a` and in history. V2 has a separate schema and must use
a separate data folder. It never upgrades a production V1 database in place.
V2 has its own schema and must use its own data folder. It never opens or upgrades a V1
database.
1. Use V1's backup facility to make a consistent SQLite copy. If copying files yourself,
stop V1 first; copying only a live WAL-mode `.db` can omit recent writes.
2. Keep the original and a verified backup untouched. Start V2 with an empty V2 volume.
3. Settings no longer shows the V1 import (its place went to restoring a full V2 backup), but
the endpoint remains. While signed in, preview the **backup copy** with
`POST /api/import/v1?preview=1` (multipart field `file`, header `X-Requested-With: studydesk`),
then import it without `?preview=1`.
4. Review the reported counts/skipped items, then explicitly import. V2 opens the submitted
SQLite file read-only and creates new IDs in one transaction. A failed import rolls back.
5. Compare class/folder/deck/quiz counts and representative questions, rationales, attempts and
scheduling. V1's passwords, provider settings and share URLs are not reused. Recreate links
deliberately. Running the import again creates another set of content.
## From V1
The importer supports V1 classes, material groups, decks/cards, quiz sets/questions/options,
completed attempts, spaced-repetition card state and activity where present. It reports
unsupported/skipped records. Older optional game features are not migrated as games.
V1 JSON quiz/flashcard exports can also be pasted/uploaded in a class's Import dialog.
V2 no longer imports V1 databases. The importer (`server/services/v1import.ts` and the
`/api/import/v1` endpoint) was removed on 2026-11-16, and Settings → Data now restores full V2
backups instead. Two ways remain for V1 content:
Portable V2 JSON exports are useful for migration between apps; full ZIP backups are needed
to preserve progress, documents, settings and credentials. See DEPLOYMENT.md for recovery.
- **V1 JSON exports.** V1 quiz and flashcard exports can be pasted or uploaded in a class's
Import dialog. This brings content only: no attempts, schedules or activity.
- **The old importer.** It is in git history, last as of commit `7125dec`. Check that commit out
into a separate working copy and run it against an empty V2 data folder. Point it at a
consistent **backup copy** of the V1 database, never the live file. Make a full V2 backup
afterwards, then restore that backup on the current version.
V1 code remains on `main` at `7bd0a9a` and in history.
## Between V2 installs
- **Full backup ZIP** (Settings → Data → Download backup, then Restore a full backup on the
other server): everything, including progress, documents, settings and AI credentials. See
DEPLOYMENT.md.
- **Portable JSON export**: classes, quizzes, flashcards and notes only, added alongside what
the other install already has.

View file

@ -1,5 +1,4 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { Hono } from 'hono';
import { bodyLimit } from 'hono/body-limit';
@ -12,7 +11,6 @@ import { createSnapshot, exportBundle, fullBackupStream, listBackups } from '../
import { createClass, createDeck, createFolder, createNote, createQuiz, draftCardToInput, draftQuestionToInput, getClass, getDeck, getQuiz, listCards, listQuestions } from '../services/content.ts';
import { createTemplate, deleteTemplate, listTemplates, resetTemplate, templateInputSchema, updateTemplate } from '../services/templates.ts';
import { restoreFullBackup } from '../services/restore.ts';
import { importV1 } from '../services/v1import.ts';
import { sessionHash } from '../auth.ts';
import { slugify } from '../util.ts';
@ -215,29 +213,6 @@ export function dataRoutes(app: AppContext): Hono {
}
});
/**
* Migration from a V1 database file. `?preview=1` only reports what would be imported. Not in
* Settings any more (full-backup restore took its place); see docs/MIGRATION.md.
*/
r.post('/import/v1', bodyLimit({ maxSize: 500 * 1024 * 1024 }), async (c) => {
const form = await c.req.parseBody().catch(() => {
throw badRequest('Upload could not be read');
});
const file = form.file;
if (!file || typeof file === 'string') throw badRequest('Choose your V1 study.db file');
const bytes = new Uint8Array(await file.arrayBuffer());
if (new TextDecoder().decode(bytes.subarray(0, 15)) !== 'SQLite format 3') throw badRequest('That file is not a SQLite database.');
const tmp = path.join(os.tmpdir(), `studydesk-v1-${Date.now()}-${Math.random().toString(36).slice(2)}.db`);
fs.writeFileSync(tmp, bytes);
try {
const preview = c.req.query('preview') === '1';
if (!preview) createSnapshot(app, 'pre-v1-import');
return c.json({ preview, summary: importV1(app, tmp, { dryRun: preview }) });
} finally {
fs.rmSync(tmp, { force: true });
}
});
/* ------------------------------ export + backup ------------------------------ */
r.get('/export/all', (c) => {

View file

@ -1,258 +0,0 @@
/**
* One-way migration from a Study Desk V1 database (the Prisma/SQLite `study.db`).
* The V1 file is opened read-only and never modified. Everything is copied into new
* rows with new ids inside one transaction, so a failed import leaves V2 untouched.
*/
import { DatabaseSync } from 'node:sqlite';
import { isFullyCorrect, scoreQuestion, type Option } from '../../shared/content.ts';
import { badRequest, type AppContext } from '../http.ts';
import { newId, now } from '../util.ts';
import { createClass, createDeck, createFolder, createQuiz, listCards, listQuestions } from './content.ts';
export interface V1Summary {
classes: number;
folders: number;
decks: number;
cards: number;
quizzes: number;
questions: number;
attempts: number;
scheduledCards: number;
activityDays: number;
skipped: string[];
}
type Row = Record<string, any>;
function toMs(v: unknown): number | null {
if (v === null || v === undefined) return null;
if (typeof v === 'number') return v > 1e11 ? v : v * 1000;
if (typeof v === 'bigint') return Number(v);
const n = Number(v);
if (Number.isFinite(n) && String(v).trim() !== '') return n > 1e11 ? n : n * 1000;
const t = Date.parse(String(v));
return Number.isNaN(t) ? null : t;
}
export function importV1(app: AppContext, file: string, opts: { dryRun?: boolean } = {}): V1Summary {
let src: DatabaseSync;
try {
src = new DatabaseSync(file, { readOnly: true });
} catch {
throw badRequest('That file could not be opened as a SQLite database.');
}
try {
const tables = new Set((src.prepare(`SELECT name FROM sqlite_master WHERE type = 'table'`).all() as Row[]).map((r) => String(r.name)));
if (!tables.has('Class') || !tables.has('Deck') || !tables.has('QuizSet')) {
throw badRequest('This does not look like a Study Desk V1 database (expected the tables Class, Deck and QuizSet).');
}
const all = (table: string, order = ''): Row[] => (tables.has(table) ? (src.prepare(`SELECT * FROM "${table}" ${order}`).all() as Row[]) : []);
const classes = all('Class', 'ORDER BY sortOrder, createdAt');
const groups = all('MaterialGroup', 'ORDER BY sortOrder, createdAt');
const decks = all('Deck', 'ORDER BY sortOrder, createdAt');
const cards = all('Flashcard', 'ORDER BY sortOrder');
const quizzes = all('QuizSet', 'ORDER BY sortOrder, createdAt');
const questions = all('Question', 'ORDER BY sortOrder');
const options = all('AnswerOption', 'ORDER BY sortOrder');
const attempts = all('QuizAttempt', 'ORDER BY completedAt');
const srStates = all('SpacedRepetitionCardState');
const activity = all('StudyActivity');
const summary: V1Summary = {
classes: classes.length,
folders: groups.length,
decks: decks.length,
cards: cards.length,
quizzes: quizzes.length,
questions: questions.length,
attempts: attempts.length,
scheduledCards: new Set(srStates.map((s) => s.flashcardId)).size,
activityDays: new Set(activity.map((a) => new Date(toMs(a.occurredAt) ?? 0).toISOString().slice(0, 10))).size,
skipped: [],
};
if (opts.dryRun) return summary;
const { db } = app;
const group = <T extends Row>(rows: T[], key: string) => {
const m = new Map<string, T[]>();
for (const r of rows) {
const k = String(r[key]);
if (!m.has(k)) m.set(k, []);
m.get(k)!.push(r);
}
return m;
};
const cardsByDeck = group(cards, 'deckId');
const questionsByQuiz = group(questions, 'quizSetId');
const optionsByQuestion = group(options, 'questionId');
const attemptsByQuiz = group(attempts, 'quizSetId');
// Keep the most-practised schedule when a card belonged to several V1 spaced-repetition sets.
const bestState = new Map<string, Row>();
for (const s of srStates) {
const cur = bestState.get(s.flashcardId);
if (!cur || Number(s.reps) > Number(cur.reps)) bestState.set(s.flashcardId, s);
}
db.tx(() => {
const classMap = new Map<string, string>();
const folderMap = new Map<string, string>();
for (const c of classes) {
const created = createClass(db, { name: String(c.name || 'Untitled class').slice(0, 120) });
classMap.set(c.id, created.id);
const t = toMs(c.createdAt);
if (t) db.run('UPDATE classes SET created_at = ? WHERE id = ?', t, created.id);
}
for (const g of groups) {
const classId = classMap.get(g.classId);
if (!classId) continue;
folderMap.set(g.id, createFolder(db, classId, String(g.name || 'Folder')).id);
}
for (const d of decks) {
const classId = classMap.get(d.classId);
if (!classId) {
summary.skipped.push(`Deck "${d.name}" (its class is missing)`);
continue;
}
const source = (cardsByDeck.get(d.id) ?? []).filter((c) => String(c.front ?? '').trim() && String(c.back ?? '').trim());
const deck = createDeck(db, {
classId,
folderId: d.groupId ? folderMap.get(d.groupId) : null,
title: String(d.name || 'Untitled deck').slice(0, 200),
description: String(d.description ?? '').slice(0, 4000),
cards: source.map((c) => ({ front: String(c.front).slice(0, 20000), back: String(c.back).slice(0, 20000) })),
});
const created = listCards(db, deck.id);
source.forEach((old, i) => {
const s = bestState.get(old.id);
const target = created[i];
if (!s || !target) return;
db.run(
`UPDATE cards SET state = ?, due = ?, stability = ?, difficulty = ?, elapsed_days = ?, scheduled_days = ?, learning_steps = ?, reps = ?, lapses = ?, last_review = ?, introduced_at = ? WHERE id = ?`,
Number(s.state) || 0,
toMs(s.due),
Number(s.stability) || 0,
Number(s.difficulty) || 0,
Number(s.elapsedDays) || 0,
Number(s.scheduledDays) || 0,
Number(s.learningSteps) || 0,
Number(s.reps) || 0,
Number(s.lapses) || 0,
toMs(s.lastReview),
toMs(s.firstReviewedAt),
target.id,
);
});
}
for (const z of quizzes) {
const classId = classMap.get(z.classId);
if (!classId) {
summary.skipped.push(`Quiz "${z.name}" (its class is missing)`);
continue;
}
const source = (questionsByQuiz.get(z.id) ?? []).filter((q) => {
const opts2 = optionsByQuestion.get(q.id) ?? [];
return String(q.prompt ?? '').trim() && opts2.length >= 2 && opts2.some((o) => o.isCorrect);
});
const quiz = createQuiz(db, {
classId,
folderId: z.groupId ? folderMap.get(z.groupId) : null,
title: String(z.name || 'Untitled quiz').slice(0, 200),
description: String(z.description ?? '').slice(0, 4000),
questions: source.map((q) => {
const opts2 = (optionsByQuestion.get(q.id) ?? []).filter((o) => String(o.text ?? '').trim()).slice(0, 10);
const correct = opts2.filter((o) => o.isCorrect).length;
return {
type: String(q.type).toUpperCase() === 'SATA' || correct > 1 ? 'multi' : 'single',
stem: String(q.prompt).slice(0, 20000),
options: opts2.map((o) => ({ text: String(o.text).slice(0, 4000), correct: Boolean(o.isCorrect) })),
rationale: String(q.rationale ?? '').slice(0, 20000),
topic: String(q.category ?? '').slice(0, 80),
};
}),
});
const created = listQuestions(db, quiz.id);
// Old question / option ids -> new, so past attempts can be replayed.
const qMap = new Map<string, (typeof created)[number]>();
const oMap = new Map<string, string>();
source.forEach((old, i) => {
const target = created[i];
if (!target) return;
qMap.set(old.id, target);
const oldOpts = (optionsByQuestion.get(old.id) ?? []).filter((o) => String(o.text ?? '').trim()).slice(0, 10);
oldOpts.forEach((o, j) => target.options[j] && oMap.set(o.id, target.options[j].id));
});
for (const a of attemptsByQuiz.get(z.id) ?? []) {
let answers: Record<string, unknown> = {};
try {
const parsed = JSON.parse(String(a.answersJson ?? '{}'));
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) answers = parsed;
} catch {
/* an unreadable attempt is still imported as a score */
}
const partial = Boolean(a.isPartialRetake);
const included = created.filter((q) => {
const oldId = [...qMap.entries()].find(([, v]) => v.id === q.id)?.[0];
return !partial || (oldId !== undefined && oldId in answers);
});
const completedAt = toMs(a.completedAt) ?? now();
const newAnswers: Record<string, { selected: string[]; checked: boolean; at: number }> = {};
for (const [oldQ, sel] of Object.entries(answers)) {
const target = qMap.get(oldQ);
if (!target || !Array.isArray(sel)) continue;
newAnswers[target.id] = { selected: sel.map((s) => oMap.get(String(s))).filter((s): s is string => Boolean(s)), checked: true, at: completedAt };
}
const attemptId = newId();
const snapshot = included.map((q) => ({ id: q.id, quizId: quiz.id, quizTitle: quiz.title, type: q.type, stem: q.stem, options: q.options as Option[], rationale: q.rationale, topic: q.topic, difficulty: q.difficulty }));
db.run(
`INSERT INTO attempts (id, quiz_id, class_id, title, mode, kind, status, settings, snapshot, answers, question_count, score, max_score, started_at, updated_at, completed_at)
VALUES (?, ?, ?, ?, 'practice', ?, 'completed', ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
attemptId,
quiz.id,
classId,
quiz.title,
partial ? 'retake' : 'full',
JSON.stringify({ shuffleQuestions: false, shuffleOptions: false, timeLimitSec: null, sataScoring: 'partial' }),
JSON.stringify({ questions: snapshot }),
JSON.stringify(newAnswers),
snapshot.length,
Number(a.score) || 0,
Number(a.maxScore) || snapshot.length,
completedAt,
completedAt,
completedAt,
);
for (const q of snapshot) {
const selected = newAnswers[q.id]?.selected ?? [];
if (selected.length === 0 && partial) continue;
const score = scoreQuestion(q, selected, 'partial');
db.run(
'INSERT OR IGNORE INTO responses (attempt_id, question_id, quiz_id, class_id, topic, score, correct, answered_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
attemptId,
q.id,
quiz.id,
classId,
q.topic,
score,
isFullyCorrect(score) ? 1 : 0,
completedAt,
);
}
}
}
// Flashcard activity keeps the study heatmap and streak intact. Quiz activity is rebuilt from the attempts above.
for (const a of activity) {
if (String(a.type) !== 'FLASHCARD') continue;
const t = toMs(a.occurredAt);
if (t) db.run(`INSERT INTO reviews (card_id, deck_id, class_id, rating, state_before, scheduled_days, mode, prev, reviewed_at) VALUES ('v1', NULL, NULL, 3, 0, 0, 'legacy', '', ?)`, t);
}
});
return summary;
} finally {
src.close();
}
}

View file

@ -1,61 +1,11 @@
import crypto from 'node:crypto';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { DatabaseSync } from 'node:sqlite';
import { describe, expect, it } from 'vitest';
import { DB, migrate } from '../server/db.ts';
import { MIGRATIONS } from '../server/migrations.ts';
import { importV1 } from '../server/services/v1import.ts';
import { createTestApp } from './helpers.ts';
describe('safe migration paths', () => {
it('previews and imports a V1 database without changing its bytes, preserving answers and schedules', async () => {
const app = await createTestApp();
const file = path.join(app.dir, 'v1-fixture.db');
const old = new DatabaseSync(file);
old.exec(`
CREATE TABLE Class (id TEXT, name TEXT, sortOrder INTEGER, createdAt INTEGER);
CREATE TABLE MaterialGroup (id TEXT, classId TEXT, name TEXT, sortOrder INTEGER, createdAt INTEGER);
CREATE TABLE Deck (id TEXT, classId TEXT, groupId TEXT, name TEXT, description TEXT, sortOrder INTEGER, createdAt INTEGER);
CREATE TABLE Flashcard (id TEXT, deckId TEXT, front TEXT, back TEXT, sortOrder INTEGER);
CREATE TABLE QuizSet (id TEXT, classId TEXT, groupId TEXT, name TEXT, description TEXT, sortOrder INTEGER, createdAt INTEGER);
CREATE TABLE Question (id TEXT, quizSetId TEXT, type TEXT, prompt TEXT, rationale TEXT, category TEXT, sortOrder INTEGER);
CREATE TABLE AnswerOption (id TEXT, questionId TEXT, text TEXT, isCorrect INTEGER, sortOrder INTEGER);
CREATE TABLE QuizAttempt (id TEXT, quizSetId TEXT, score REAL, maxScore INTEGER, answersJson TEXT, isPartialRetake INTEGER, completedAt INTEGER);
CREATE TABLE SpacedRepetitionCardState (flashcardId TEXT, state INTEGER, reps INTEGER, due INTEGER, stability REAL, difficulty REAL, scheduledDays INTEGER, learningSteps INTEGER, lapses INTEGER, lastReview INTEGER, firstReviewedAt INTEGER, elapsedDays INTEGER);
INSERT INTO Class VALUES ('c','Legacy class',0,1700000000000);
INSERT INTO MaterialGroup VALUES ('g','c','Legacy group',0,1700000000000);
INSERT INTO Deck VALUES ('d','c','g','Legacy deck','',0,1700000000000);
INSERT INTO Flashcard VALUES ('f','d','Term','Definition',0);
INSERT INTO QuizSet VALUES ('z','c','g','Legacy quiz','',0,1700000000000);
INSERT INTO Question VALUES ('q','z','MULTIPLE_CHOICE','Pick A','Reason','Topic',0);
INSERT INTO AnswerOption VALUES ('a','q','A',1,0),('b','q','B',0,1);
INSERT INTO QuizAttempt VALUES ('attempt','z',1,1,'{"q":["a"]}',0,1700000000000);
INSERT INTO SpacedRepetitionCardState VALUES ('f',2,7,1800000000000,15,5,14,0,1,1700000000000,1600000000000,12);
`);
old.close();
const digest = () => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
const before = digest();
try {
expect(importV1(app.ctx, file, { dryRun: true })).toMatchObject({ classes: 1, quizzes: 1, cards: 1, attempts: 1 });
expect(app.ctx.db.get<{ n: number }>('SELECT count(*) AS n FROM classes')?.n).toBe(0);
importV1(app.ctx, file);
expect(digest()).toBe(before);
const classes = (await app.api('GET', '/api/classes')).body.classes;
expect(classes[0].name).toBe('Legacy class');
const card = app.ctx.db.get<{ reps: number; due: number; elapsed_days: number }>('SELECT reps, due, elapsed_days FROM cards');
expect(card).toMatchObject({ reps: 7, due: 1800000000000, elapsed_days: 12 });
const attempt = app.ctx.db.get<{ id: string }>('SELECT id FROM attempts')!;
const result = (await app.api('GET', `/api/attempts/${attempt.id}`)).body;
expect(JSON.stringify(result)).toContain('Pick A');
expect(app.ctx.db.get<{ score: number }>('SELECT score FROM responses')?.score).toBe(1);
expect(app.ctx.db.all('PRAGMA foreign_key_check')).toEqual([]);
} finally {
app.close();
}
});
it('backs up before a schema change and rolls back a failed migration', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'studydesk-migration-test-'));
const db = new DB(path.join(dir, 'test.db'));