From 099c53badfda5326aacab9cbcfd173dc2ee4de9b Mon Sep 17 00:00:00 2001 From: Elijah Date: Thu, 16 Jul 2026 18:40:12 -0700 Subject: [PATCH] Update architecture baseline for sync, sharing, and gateways --- AGENTS.md | 2 + Drive_v2_final_plan.md | 212 ++++++++++-------- docs/adr/0008-sync-capable-server-contract.md | 45 ++++ ...0009-external-protocol-gateway-boundary.md | 25 +++ docs/adr/README.md | 3 +- docs/architecture.md | 11 +- docs/threat-model.md | 4 +- 7 files changed, 200 insertions(+), 102 deletions(-) create mode 100644 docs/adr/0008-sync-capable-server-contract.md create mode 100644 docs/adr/0009-external-protocol-gateway-boundary.md diff --git a/AGENTS.md b/AGENTS.md index e8cd63d..be70403 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,6 +18,8 @@ - Every metadata mutation and its change event must commit in one PostgreSQL transaction. - Committed blobs are immutable. All transfers stream and all sizes/offsets use 64-bit types. - Do not add endpoints outside `api/openapi.yaml` or queries outside `db/queries/`. +- Preserve the baseline/change guarantees in ADR-0008; cursors, scoped projections, and resulting-state events are one contract. +- External gateways follow ADR-0009 and may never access managed objects or repositories directly. - Do not edit generated files directly. Regenerate them and commit source and output together. - Do not add dependencies or change architecture without owner approval and an ADR. - Keep frontend features isolated. Never create a global event bus, central extension switch, or all-purpose file-manager component. diff --git a/Drive_v2_final_plan.md b/Drive_v2_final_plan.md index 199dad8..0d09582 100644 --- a/Drive_v2_final_plan.md +++ b/Drive_v2_final_plan.md @@ -1,6 +1,6 @@ # Drive v2 — Final Architecture and Implementation Plan -Status: approved baseline. This document is the source of truth until superseded by ADRs in the repository. It is written for both human and AI implementers. Normative keywords MUST, MUST NOT, SHOULD, and MAY follow RFC 2119 meaning. Section numbers are stable and may be referenced from agents.md, ADRs, and commit messages. +Status: approved baseline, updated 2026-07-16; Phase 0 complete. This document is the source of truth until superseded by ADRs in the repository. It is written for both human and AI implementers. Normative keywords MUST, MUST NOT, SHOULD, and MAY follow RFC 2119 meaning. Section numbers are stable and may be referenced from `AGENTS.md`, ADRs, and commit messages. --- @@ -21,7 +21,7 @@ Deployment target (fixed, not generic): ## 2. Non-Negotiable Invariants -These are the promises the system makes. The automated invariant test suite (Section 14) encodes them. AI implementers MUST NOT weaken, bypass, or modify these without a human-approved ADR. +These are the promises the system makes. Focused automated tests (Section 13) encode them. AI implementers MUST NOT weaken, bypass, or modify these without a human-approved ADR. - I1. Committed metadata MUST never reference unavailable, incomplete, or unverified content. - I2. Managed content objects are immutable once committed. Nothing modifies a committed object in place. New content means a new object and a new revision. @@ -32,15 +32,19 @@ These are the promises the system makes. The automated invariant test suite (Sec - I7. Only the storage adapter touches managed array paths. Only application services coordinate database plus storage mutations. Enforced by CI (Section 5). - I8. Errors are never silently discarded. - I9. Directory listing and search MUST never scan the array at request time. -- I10. The server never applies silent last-write-wins. Stale mutations fail with 409 and current server state. +- I10. The server never applies silent last-write-wins. Failed HTTP revision preconditions return 412 with current server state; semantic conflicts return 409. - I11. In-flight (uncommitted) uploads MAY be lost on unclean shutdown. Committed data MUST NOT be. Crash recovery protects commits, not transfers. -- I12. GC MUST NOT remove any object referenced by a live node, an active replacement-recovery record, an active editor session, or an incomplete commit intent. +- I12. GC MUST NOT remove any object referenced by a live node, active export lease, replacement-recovery record, editor session, or incomplete commit intent. +- I13. A start cursor plus complete recursive enumeration and subsequent change replay MUST be sufficient to reconstruct current server state without missing a committed mutation. +- I14. Every mutation source MUST use the same application services, revision checks, audit path, and transactional change feed. +- I15. Replaying a request with the same idempotency key and request fingerprint MUST NOT perform its mutation twice; reusing the key for different content MUST fail. +- I16. External protocol gateways MUST NOT access managed object storage or mutate repository tables directly. +- I17. Client-supplied hashes and modification times are comparison aids only; they MUST NOT bypass server verification, authorization, or revision preconditions. ## 3. Capacity Contract - No intentionally low application-level file-size limit. Default administrative upload ceiling: 1 TiB, raisable. -- Stable releases are qualified with real 300 GB-class upload, download, checksum, restart, and ZIP64 tests. Larger files are supported up to the configured ceiling but not separately certified. -- One-time Phase 0 validation at 10 million synthetic nodes and 500,000-descendant subtrees proves the schema and query plans. These are NOT recurring release gates (Section 14). +- All transfer and hierarchy paths are designed for the stated file sizes and stream with bounded memory. Large-scale benchmarks and 300 GB qualification are deferred until representative product workflows exist and are not v1 release gates. - Unraid share minimum free space MUST be set to at least 400 GB (above the largest expected file class). - Free-space enforcement happens at upload-session creation (declared size vs. eligible-disk free space plus safety margin) and per accepted chunk. Finalization adds no bytes; a finalization-time space check is redundant and MUST NOT be the primary control. @@ -62,17 +66,18 @@ These are the promises the system makes. The automated invariant test suite (Sec - Required server-side bulk importer (Section 8.3). - OpenAPI-documented `/api/v1`; installable PWA (Section 10.4). - Previews: images (JPEG, PNG, GIF, WebP, SVG, BMP), ranged audio/video, PDF (PDF.js), bounded text/Markdown/CSV/JSON/logs, source code with highlighting, lazy viewport thumbnails. +- Public link sharing: expiry, optional password, download limits, revocation, and abuse protection. +- OnlyOffice viewing, editing, creation, and templates through the integration model in Section 11.1. +- A complete sync-ready server contract: recursive baseline, durable changes, client attribution, capability negotiation, hashes, resumable transfer, and scoped credentials. The desktop client itself is deferred. - Light/dark themes. -### 4.2 Post-v1 native modules (order-flexible; see Section 15) +### 4.2 Post-v1 modules and compatibility integrations - ZIP/ZIP64 and 7z archive viewing; durable zip/unzip jobs. - STL and 3MF 3D preview. -- Public link sharing (expiry, optional password, download limits, revocation). -- OnlyOffice viewing/editing/creation/templates (Section 11). - Calendar with ICS subscription feeds (Section 11.2). - Mirrored-folder desktop sync client. -- WebDAV: not planned; a read-only facade over application services MAY be added later (ADR recorded). No third-party plugin system, ever; extensions are native compiled modules. +- Writable WebDAV compatibility gateway and an optional native rclone backend (ADR-0009). Rsync is not planned. No third-party plugin system, ever; extensions are native compiled modules. ## 5. Architecture @@ -104,7 +109,7 @@ Rationale: AI implementers respect failing checks with near-perfect reliability React, TypeScript (strict), Vite, TanStack Router, TanStack Query. - TypeScript API client generated from OpenAPI; drift check in CI. -- Feature modules: auth, files, uploads, search, trash, replacements, previews, jobs, settings; later sharing, office, calendar. +- Feature modules: auth, files, uploads, search, trash, replacements, sharing, office, previews, jobs, settings; later calendar. - Server state in TanStack Query only; a small UI store for selection/view state; upload-session references persisted in IndexedDB. - Preview renderers lazy-loaded, selected by server-declared capability, never by a central extension switch. - Virtualized large lists. Accessible primitives, keyboard interaction, shared design tokens. @@ -154,9 +159,15 @@ Content is logically addressed by SHA-256 in PostgreSQL. Physical objects use op ### 6.4 Core tables -`users`, `sessions`, `api_tokens`, `nodes`, `blobs`, `upload_sessions`, `commit_intents`, `replacement_recovery`, `changes`, `jobs`, `preview_assets`, `audit_events`, `settings`, `import_runs`, `import_items`. Later: shares, archive_indexes, calendar tables, editor_sessions. Module tables MUST NOT add columns to core file tables. +`users`, `sessions`, `api_tokens`, `credential_root_grants`, `client_instances`, `idempotency_records`, `nodes`, `blobs`, `upload_sessions`, `commit_intents`, `replacement_recovery`, `changes`, `jobs`, `preview_assets`, `audit_events`, `settings`, `import_runs`, `import_items`, `shares`, `editor_sessions`, `export_leases`, `export_items`. Later module tables include archive indexes, calendar data, and WebDAV locks. Module tables MUST NOT add columns to core file tables. -`nodes`: UUIDv7 id, owner, parent, name, case-folded name key, type, current blob id, size, MIME, monotonic revision, created/modified/trashed timestamps. +`nodes`: UUIDv7 id, owner, parent, name, case-folded name key, type, current blob id, size, MIME, monotonic node revision, monotonic content revision, logical content modification time, server metadata update time, created time, and trash-root time. Node revision changes for any meaningful mutation; content revision changes only when committed file content changes. SHA-256 and blob identity remain authoritative content identities. + +`changes`: monotonic sequence, event ID/type, schema version, node ID and revisions, subtree effect, immutable resulting-node snapshot or deletion tombstone, previous location/trash diagnostics where applicable, mutation origin, and commit time. + +`idempotency_records`: authenticated owner/token/client identity, key, request fingerprint, operation type, state, stable response reference, and expiry. Records persist across process restarts. + +`client_instances`: display name, type, platform, software/protocol versions, last seen time, and revocation time. API tokens MAY bind to a client instance and subtree/action grants. ## 7. File and Hierarchy Semantics @@ -179,7 +190,7 @@ Content is logically addressed by SHA-256 in PostgreSQL. Physical objects use op - Trash: marks only the root as a trash root. Descendants become invisible via ancestor visibility, not row updates. O(1). - Restore: clears the root trash state, resolves root-name conflicts. O(1). - Permanent subtree delete: durable background job. Blob reclamation is separate scheduled GC. -- Search returns indexed candidates then applies batched ancestor-visibility filtering. Direct node access checks the ancestor chain so stale IDs cannot reach descendants of trashed roots. This filtered search is the performance-critical query in the system; Phase 0 validates it explicitly. +- Search returns indexed candidates then applies batched ancestor-visibility filtering. Direct node access checks the ancestor chain so stale IDs cannot reach descendants of trashed roots. Optimize this query when realistic data demonstrates a bottleneck. ### 7.4 Replacement recovery @@ -192,8 +203,8 @@ Content is logically addressed by SHA-256 in PostgreSQL. Physical objects use op ### 8.1 Resumable uploads (TUS-compatible) -- 64 MiB default chunks, sequential, exact offset/length enforcement. -- Session record persists destination, declared size, offset, expiry. +- The Drive client defaults to 64 MiB sequential chunks. The server accepts bounded TUS chunks and enforces exact offset/length semantics without requiring one universal client chunk size. +- Session records persist create/replace target, expected revision, declared size and optional SHA-256, logical content mtime, origin client/operation, conflict policy, accepted offset, state, and expiry. - Space check at session creation and per chunk (Section 3). - Survives graceful restarts and connection interruptions (the common case: closed tab, WiFi drop, container update). - Crash posture (I11): the incremental hash checkpoint is NOT required to survive unclean shutdown. After a crash, recovery MAY discard incomplete partials or re-hash them from disk; failing the upload is acceptable. Do not engineer durability for uncommitted transfers. @@ -206,7 +217,7 @@ Content is logically addressed by SHA-256 in PostgreSQL. Physical objects use op 4. On completion: verify length, finalize SHA-256, fsync object and parent directory as supported. 5. Create commit intent. 6. Promote `.partial` → `.blob` (same directory). -7. One PostgreSQL transaction: insert-or-reuse blob, create/update node, increment revision, add replacement-recovery record if overwrite, emit change event, complete intent. +7. Recheck target, revision, parent, and name preconditions. One PostgreSQL transaction then inserts-or-reuses the blob, creates/updates the node, increments node/content revisions as applicable, adds replacement recovery for overwrite, emits the attributed change event, and completes the intent. 8. Startup/periodic recovery workers finish or roll back incomplete intents (this protects I1/I11 and MUST run on every start). 9. Redundant deduplicated objects deleted only after the committed blob is confirmed available. @@ -233,22 +244,46 @@ TUS-per-file overhead is accepted for v1 and measured, not hidden. Post-v1 candi OpenAPI 3.1 is the public source of truth and lives in the repository. -Endpoints: `/api/v1/setup`, `/sessions`, `/api-tokens`, `/nodes`, `/nodes/{id}`, `/nodes/{id}/children`, `/nodes/{id}/content`, `/uploads`, `/search`, `/trash`, `/replacements`, `/changes`, `/jobs`, `/previews`; `/health/live`, `/health/ready`, `/metrics`. Later: `/shares`, `/office`, `/calendar`. +Stable-v1 endpoint families: `/api/v1/setup`, `/sessions`, `/api-tokens`, `/client-instances`, `/capabilities`, `/nodes`, `/nodes/{id}`, `/nodes/{id}/children`, `/nodes/{id}/descendants`, `/nodes/{id}/content`, `/uploads`, `/search`, `/trash`, `/replacements`, `/changes`, `/jobs`, `/previews`, `/shares`, and `/office`; `/health/live`, `/health/ready`, and `/metrics`. Calendar endpoints are added only when the deferred Calendar module is implemented. Rules: - Opaque keyset cursors; no offset pagination on large collections. - RFC 9457 problem responses with stable machine-readable error codes. -- `ETag` + `If-Match` or `expected_revision` on mutations; stale → `409` with current metadata (I10). -- `Idempotency-Key` on create, finalize, copy, move, import, and destructive requests. -- Standard `Range`/`Content-Range`/`Content-Disposition` and cache validators. -- Request IDs propagated through logs and job records. -- `/changes?cursor=` is the durable authority; SSE provides hints only. Change/tombstone retention: 365 days; expired cursor → `410 Gone` → full rebaseline. +- `ETag` + `If-Match` or `expected_revision` on mutations. Failed representation preconditions return `412`; name/hierarchy/domain conflicts return `409` (I10 and ADR-0005). +- `Idempotency-Key` on create, finalize, copy, move, import, restore, trash/delete, and other retryable mutations. The server persists the request fingerprint and original result (I15). +- Authentication establishes a protocol-neutral mutation context: actor, credential, validated client instance, operation ID, idempotency key, and request ID. REST, importer, OnlyOffice, and future gateways pass this same context to application services. +- Standard `Range`/`Content-Range`/`Content-Disposition`, cache validators, and RFC 9530 `Content-Digest`/`Repr-Digest` where applicable. SHA-256 is also present in JSON node metadata. +- API tokens can be restricted by action and root node. External credentials default to trash rather than permanent deletion and do not gain access outside their granted subtree. +- Optional declared SHA-256 and size are accepted during upload creation. After target authorization and precondition checks, the server MAY perform a no-op or attach an already verified owner blob through the normal commit service; newly transferred bytes are always server-verified (I17). - Folder move/trash/delete events apply to the subtree rooted at the referenced ID (one event, Section 7.3). -### 9.1 Sync conformance mirror (permanent tool) +### 9.1 Capability negotiation -Phase 2 builds a one-way download mirror: baseline the tree, poll `/changes`, download content, apply rename/move/trash/delete, persist cursor, survive restart, handle tombstones, folder moves, forced rebaseline, verify checksums. It stays in the repository forever as the sync-contract regression harness. Full bidirectional sync remains Phase 8 and consumes the contract this tool proves. +`GET /api/v1/capabilities` reports API and sync protocol versions, minimum compatible client protocol, hash algorithms, naming rules, timestamp precision, change retention, administrative upload ceiling, and supported features such as recursive baseline, change cursors, resumable upload, server-side move/copy, logical content mtimes, trash, replacement recovery, and root-scoped credentials. Clients MUST use capabilities instead of inferring behavior from the server software version. + +### 9.2 Consistent baseline protocol + +1. Obtain opaque cursor `C0` from `GET /api/v1/changes/start-cursor`. +2. Enumerate the selected root with keyset-paginated `GET /api/v1/nodes/{rootId}/descendants`, returning stable node/parent IDs and requested metadata fields. +3. Replay `/api/v1/changes?cursor=C0` until caught up. +4. Persist the cursor returned after the last processed event. + +Concurrent mutations may be observed both in enumeration and replay, so duplicate observations are expected. This sequence MUST not miss committed mutations while `C0` remains within retention (I13). It does not hold a long-running PostgreSQL snapshot. + +### 9.3 Change delivery contract + +- `/changes` is the durable authority; SSE only signals that polling may find work. +- Events are ordered by monotonic sequence and consumers treat delivery as at least once. Page cursors identify the position after the final returned event. +- Each event carries a versioned resulting-node snapshot or deletion tombstone, node/content revisions, subtree effect, origin client/operation, and useful previous-location diagnostics. Resulting state is authoritative. +- Processing by node ID plus revision is idempotent; consumers tolerate duplicate events, already-applied revisions, repeated subtree effects, and restart between side effect and cursor commit. +- A subtree-scoped credential receives an authorization-safe projection of the global sequence. Moves into its root appear as current snapshots; moves out appear as tombstones/removals; unrelated events are omitted while the cursor still advances. Baseline enumeration and change replay use the same root anchor and visibility rules. +- Change/tombstone retention is 365 days. An expired cursor returns `410 Gone` with stable code `change_cursor_expired`; consumers preserve pending local work, establish a new baseline, and reconcile rather than discarding it. +- Every mutation source emits through this same transactional mechanism (I3 and I14). + +### 9.4 Sync conformance mirror (permanent tool) + +Phase 2 builds a one-way download mirror: baseline the tree, poll `/changes`, download content, apply rename/move/trash/delete, persist cursor, survive restart, handle tombstones, folder moves, forced rebaseline, and verify checksums. Its focused contract tests cover mutation during enumeration, duplicate events, interrupted rebaseline, cursor expiry with pending local state, restart before cursor commit, queued descendant work during folder moves, and originating-client acknowledgement. It stays in the repository as the regression harness. A future bidirectional client consumes the contract it proves. ## 10. Capability Providers @@ -297,13 +332,13 @@ Requirements: - Calendars, events, all-day and timed events, IANA time zones, recurrence with exceptions and moved instances, reminders, attachments, ICS import/export. - Recurrence MUST use a maintained RFC 5545 library (selected by ADR with conformance fixtures covering DST boundaries, EXDATE/RDATE, moved instances). Hand-rolled RRULE evaluation is PROHIBITED. - Read-only ICS subscription feeds: `/api/v1/calendar/feeds/.ics`. Tokens are high-entropy, stored hashed, scoped to one calendar, revocable, rotatable; UI warns that URL possession grants read access. Writes remain via UI/API. No CalDAV. -- PWA calendar views and notifications where browser support permits. +- PWA calendar views. Reminder definitions may be stored, but notification delivery is deferred from the initial Calendar release. ## 12. Security, Reliability, Operations ### 12.1 Security -- Argon2id with versioned parameters; server-side revocable sessions in `HttpOnly`/`Secure`/`SameSite` cookies; CSRF protection on cookie-authenticated mutations; hashed scoped API tokens; no long-lived browser JWTs. +- Argon2id with versioned parameters; server-side revocable sessions in `HttpOnly`/`Secure`/`SameSite` cookies; CSRF protection on cookie-authenticated mutations; hashed action- and subtree-scoped API tokens; client-instance binding and independent revocation; no long-lived browser JWTs. - Trusted-proxy configuration; rate limiting on validated client IP; strict CSP and security headers; MIME-sniff protection; symlink rejection; archive protections (10.3); OnlyOffice callback auth and SSRF protection (11.1). - Application-level encryption at rest: not included (use Unraid/disk encryption if desired), preserving dedup and recovery practicality. @@ -330,108 +365,86 @@ Structured JSON logs (request/user/node/upload/job IDs). Prometheus metrics: req ## 13. Testing Policy (Recalibrated) -Principle: a test earns permanence if it protects against silent data corruption, or if it is automated and free to run forever. Manual drills against loud, recoverable failures are cut. +Testing is proportionate to a private single-owner application. Early work favors normal unit tests and focused integration tests over synthetic scale, endurance, or exhaustive crash matrices. -Permanent, automated (CI and release gates): +- Go and TypeScript unit tests cover naming, hierarchy, revisions, idempotency, change ordering, deduplication, conflicts, replacement recovery, and UI behavior. +- Focused PostgreSQL/storage integration tests cover the commit protocol, mutation-plus-change transaction, retry behavior, and core API operations. +- The sync conformance mirror permanently covers the baseline and delivery guarantees in Section 9. It uses ordinary fixtures, not massive generated trees. +- A limited Playwright suite covers setup, login/2FA, upload, CRUD, conflicts, sharing, OnlyOffice, previews, trash, recovery, and session expiry. Include an accessibility smoke test for login and the primary file workflow. +- Security tests cover traversal, symlinks, malformed ranges, CSRF, token/grant revocation, proxy IP handling, SSRF, unauthorized access, and gateway boundary enforcement. +- Import tests cover deduplication, collisions, invalid names, changed sources, read-only sources, empty folders, and a normal restart. +- Before production adoption, perform a clean Unraid install, normal upgrade/restart, representative large-file transfer, importer run, backup, and one restore. -- Unit/property tests: naming, hierarchy, revisions, dedup, conflicts, replacement recovery, cursor ordering, GC eligibility. -- Integration tests for every mutation (PostgreSQL + storage). -- Failure injection around partial write, fsync, promotion, transaction commit, event emission, cleanup. -- Process/container kill tests at every commit boundary, including on the real shfs path during development. These are the crash tests that matter. -- Restart tests during upload, import, finalization, preview, archive jobs, scrub, GC, backup. -- Sync-conformance harness runs (Section 9.1). -- Playwright critical flows: setup, login/2FA, upload, pause/resume, CRUD, conflicts, previews, search, trash, replacement restore, jobs, session expiry. One inexpensive accessibility smoke test on login + primary file workflow. Automated Chromium only; manual release check on Brave desktop and Android Chromium. -- Security tests: traversal, symlinks, malformed ranges, CSRF, token revocation, proxy IPs, archive bombs, SSRF, unauthorized access. -- Import tests: restart, dedup, collisions, invalid names, changed sources, read-only sources, empty folders. - -Release gates using real data: - -- Real 300 GB upload/download with checksum verification; real 300 GB ZIP64 browse without full extraction or unbounded memory (once the archive module ships). -- pgBackRest restore drill per stable release. - -One-time Phase 0 validation (NOT recurring gates): - -- 10-million-node directory/search benchmarks (pagination < 250 ms p95, indexed search < 1 s p95 on the reference host). -- 500,000-descendant move/trash/restore/search/delete semantics, including ancestor-visibility-filtered search performance. -- Same-directory promotion protocol exercised on the real `/mnt/user` array path under process/container termination. -- PostgreSQL-on-btrfs (CoW) benchmark on the real cache. - -Explicitly NOT required: - -- Recurring power-loss testing. OPTIONAL curiosity in a disposable Unraid VM (trial key, small virtual parity array) if ever desired; NEVER on the production server. -- Hash-checkpoint durability across unclean shutdown (I11). -- 1 TiB or 16 TiB certification tiers; 10M-node recurring benchmarks; full browser matrix; Storybook. +Deferred until real workloads justify them: 10-million-node benchmarks, 500,000-node subtrees, 300 GB certification, PostgreSQL/btrfs benchmarks, repeated process-kill matrices, endurance suites, and full browser coverage. Add targeted stress or recovery tests when actual profiling or failures identify a risk. Hash-checkpoint durability for uncommitted uploads remains explicitly unnecessary (I11). ## 14. AI Implementation Policy Most code will be written by AI agents (multiple models over time). The repository, not chat history, is the agents' world. -- This plan's contracts live IN the repository: OpenAPI spec, SQL schema + sqlc queries, ADRs, per-module docs. agents.md references them by path and section. +- This plan's contracts live IN the repository: OpenAPI spec, SQL schema + sqlc queries, ADRs, and per-module docs. `AGENTS.md` references them by path and section. - Contract-first is anti-hallucination infrastructure: agents cannot invent endpoints (generated client + drift check) or columns (sqlc compile failure). The human owns the OpenAPI spec, schema, and sqlc queries as reviewed artifacts; agents implement between them. Own the interfaces, delegate the implementations. - Architecture boundaries are lint failures, not guidelines (Section 5.2). - The invariant test suite (encoding Section 2) is AGENT-IMMUTABLE: it lives in a dedicated directory that agents may not modify without explicit human approval; any diff touching it requires human review. Making a red test green by editing the test is the canonical failure mode this prevents. - Every AI change: small, independently testable, reviewed against the invariant it implements. Dependency APIs verified against primary documentation; no invented methods. No new dependencies without approval. No unrelated "misc fixes" in a change. -- Every data mutation ships with success, failure, restart, cleanup, and idempotency tests. +- Data mutations ship with focused success, conflict/failure, and idempotency coverage appropriate to their risk. Add restart/cleanup cases where durable work or cross-system commits require them. - Architectural changes require an ADR before code. -- A feature is complete only when errors, cancellation, restart, observability, and recovery behavior are covered. -- agents.md (TBD) stays short and operational: build/test/lint commands, boundary rules, the Section 2 invariants restated as MUST, forbidden patterns, pointer to ADRs. Long instruction files degrade agent compliance. +- A feature is complete when its expected errors, authorization, cancellation where applicable, observability, and recovery behavior are defined and tested proportionately. +- `AGENTS.md` stays short and operational: build/test/lint commands, boundary rules, Section 2 invariants, forbidden patterns, and ADR pointers. Long instruction files degrade agent compliance. ## 15. Implementation Phases -Phases 5-7 are order-flexible after Phase 4; the calendar module has no dependency on archives or sharing and MAY be pulled earlier given heavy real-world use. +### Phase 0 — Repository and development foundation (complete) -### Phase 0 — Validate foundations +- Repository conventions, `AGENTS.md`, ADR process, threat model, dependency policy, Go/React module boundaries, OpenAPI/sqlc skeletons, Compose environment, Forgejo CI, and versioned container build. +- Minimal health/setup-status server and frontend shell with boundary, unit, contract, and image smoke checks. -- Freeze and inventory the old repository. Create new repo, CI (lint boundaries from day one), ADR process, threat model, UI direction, OpenAPI skeleton, Compose environment. -- Run the one-time validations from Section 13: 10M-node benchmark, 500k-descendant semantics, promotion protocol on the real array path under kill testing, Postgres-on-btrfs benchmark, real 300 GB streaming + hashing prototype. -- Exit when storage recovery and metadata performance meet documented targets. +### Phase 1 — Core domain, persistence, authentication, and storage -### Phase 1 — Domain, database, authentication +- Migrations/repositories for hierarchy, node/content revisions and timestamps, replacement recovery, attributed changes, persistent idempotency, client instances, scoped credentials, jobs, settings, and audit events. +- Owner setup, sessions, recovery codes, TOTP, API tokens, CLI recovery, CSRF, trusted proxy, health, metrics, and structured logs. +- Storage objects, commit intents, promotion/deduplication, recovery sweep, resource leases, reference tracking, and GC eligibility. +- Exit when focused unit/integration tests demonstrate that committed metadata cannot reference incomplete content and mutations cannot detach from their change event. -- Migrations, repositories, node hierarchy, naming, revisions, replacement recovery, change feed, jobs, sessions, 2FA, API tokens, health, metrics, audit events. -- Storage objects, commit intents, recovery sweep, reference tracking, GC eligibility. -- Exit when failure injection proves I1: metadata can never reference unavailable objects. +### Phase 2 — Transfers, importer, and sync-ready server contract -### Phase 2 — Transfers, importer, sync contract +- TUS uploads with pause/resume/cancel, offset reconciliation, optional hash preflight, explicit commit, range/bulk/folder downloads, and logical content mtime. +- Start cursor, recursive enumeration, complete change polling, SSE hints, capability negotiation, tombstones, and permanent conformance mirror per Section 9. +- Importer dry-run, resumable run state, conflicts, deduplication, reports, and representative real-corpus ingestion. +- Search, favorites, trash, replacement recovery, backup tooling, manifest export, and practical integrity checks. +- Exit when ordinary interruption/retry scenarios converge without duplicate mutation or missed change. -- TUS uploads with cancellation/resumption; range/bulk/folder downloads; search; favorites; trash; jobs; integrity checks; backup tooling; manifest export. -- Importer with dry-run modes; ingest the real few-hundred-GB corpus as the acceptance test. -- Permanent one-way mirror conformance tool. -- Real 300 GB upload/download/restart/checksum validation. -- Exit when upload, import, and change-feed recovery survive repeated interruption. +### Phase 3 — File operations and durable maintenance -### Phase 3 — Web application and PWA +- Complete create, rename, move, copy, trash, restore, permanent delete, replacement recovery, favorites, search, and batch operations. +- Large subtree copies/deletes, export snapshots, scrub, and GC as persistent checkpointed jobs with progress/cancellation and safe cleanup. +- Exit when all normal file-management operations and conflicts are functional through the API. -- File manager, uploads, navigation, search, trash, replacement recovery UI, settings, jobs, activity, storage dashboards. -- Core preview providers and lazy viewport thumbnails. -- Installable PWA with feature-detected Share Target. -- Responsive, keyboard, and end-to-end tests. First release candidate. +### Phase 4 — Core web application -### Phase 4 — Stable v1 hardening +- Setup/authentication, responsive file navigation, list/grid views, keyset pagination, virtualization, keyboard navigation, selection, uploads, conflict dialogs, search, favorites, trash, replacements, and operational dashboards. +- Maintain feature import boundaries, component-size limits, accessibility, responsive Brave/Android layouts, and light/dark themes. +- Exit when day-to-day file management works without relying on previews or office integration. -- Full automated suite green; real 300 GB gates; scheduled scrub/GC behavior on spinning disks; security review; container scanning; upgrade test; isolated restore drill. -- Publish Unraid, proxy, container stop-timeout, UPS, backup, restore, importer, capacity, and troubleshooting docs. -- Release after a clean Unraid install and restore succeed. +### Phase 5 — Stable-v1 sharing and OnlyOffice -### Phase 5 — Archives and 3D +- Public file/folder links with passwords, expiry, limits, revocation, throttling, audit, and snapshot-backed folder downloads. +- OnlyOffice scoped sessions, revision-bearing keys, JWT callbacks, conflict-copy saves, GC pinning, blank documents, and templates. -- ZIP/ZIP64/7z providers per Section 10.3; durable zip/unzip jobs; STL/3MF providers with Web Worker parsing and geometry limits. -- Test malformed archives, solid-7z cost handling, bombs, huge entry counts, oversized models, real 300 GB ZIP64. +### Phase 6 — Stable-v1 previews and PWA -### Phase 6 — Sharing and OnlyOffice +- Capability-driven image/media/PDF/text/source previews, viewport-driven thumbnails, bounded cache, resource/security limits, installable PWA, and Android Share Target. -- Public links (expiry, passwords, limits, revocation, abuse protection). -- OnlyOffice per Section 11.1: scoped sessions, revision-bearing document keys, JWT callbacks, conflict-copy saves, GC pinning, templates. +### Phase 7 — Practical hardening and stable-v1 release -### Phase 7 — Calendar +- Standard automated suite, focused security review, container scan, clean Unraid install, upgrade/restart, representative large-file workflow, importer run, backup, and one restore. +- Publish Unraid, proxy, setup order, OnlyOffice, shutdown, backup/restore, importer, capacity, and troubleshooting documentation. -- Schema, recurrence via maintained library, ICS import/export, hashed-token subscription feeds, PWA calendar views, reminders/notifications where supported. +### Deferred modules -### Phase 8 — Desktop sync - -- Shared Go sync engine on the proven change-feed contract; Wails desktop packaging; local SQLite state; filesystem watching; selective sync; resumable transfers; bandwidth limits; autostart; conflict copies. -- Windows/macOS/Linux naming and multi-device conflict validation. -- Wails is desktop-only. Native Android remains indefinitely deferred; the PWA covers mobile use. +- Desktop mirror client: standalone three-way reconciliation engine with capability-aware remote/local adapters, local SQLite state, watcher-as-hint plus scans, whole-file resumable transfers, deterministic conflict copies, and a thin Wails UI. On-demand placeholder/stream mode is a separate OS-specific product. +- Calendar: isolated schema/API, maintained RFC 5545 recurrence library, ICS feeds, PWA views, and no initial reminder delivery. +- ZIP/ZIP64/7z and STL/3MF providers with bounded durable work. +- Writable WebDAV gateway and optional native rclone backend per ADR-0009. Neither is the official live-sync engine. ## 16. Fixed Assumptions and Decision Log Seeds @@ -439,5 +452,6 @@ Phases 5-7 are order-flexible after Phase 4; the calendar module has no dependen - Drive exclusively owns managed objects after import; external writes into managed storage unsupported; hardlinks into managed storage prohibited. - Latest content is canonical; 7-day replacement window; 30-day trash. - PostgreSQL, config, previews on the exclusive btrfs cache; objects and recovery data on the array share via shfs. -- WebDAV not planned but not forbidden (ADR). No plugin system. No CalDAV. -- Initial ADRs to record: ADR-001 Postgres-on-btrfs CoW (Section 6.3), ADR-002 opaque object IDs + same-directory promotion (6.2), ADR-003 crash posture: commits protected, transfers expendable (I11), ADR-004 recurrence library selection, ADR-005 WebDAV stance, ADR-006 agent-immutable invariant tests. +- REST and application services are canonical. Future clients and native rclone use REST; protocol gateways translate into the same services and never touch managed objects directly. +- Rsync support and direct SMB/SFTP access to managed object storage are not planned. Writable WebDAV is an optional post-v1 gateway. No plugin system. No CalDAV in the planned Calendar module. +- The accepted ADR index in `docs/adr/README.md` is authoritative. New architectural decisions require a new ADR rather than edits to accepted ADR history. diff --git a/docs/adr/0008-sync-capable-server-contract.md b/docs/adr/0008-sync-capable-server-contract.md new file mode 100644 index 0000000..8bdffe0 --- /dev/null +++ b/docs/adr/0008-sync-capable-server-contract.md @@ -0,0 +1,45 @@ +# ADR-0008: Sync-capable server contract + +Status: Accepted + +## Context + +The desktop client is deferred, but the stable-v1 API must support a client that maintains a local replica without missing concurrent mutations or silently overwriting a newer revision. The previous plan named a change feed and one-way mirror but did not define a consistent initial baseline, delivery semantics, mutation attribution, or the difference between content and metadata changes. + +## Decision + +Drive's REST API and application services are the canonical server interface. The server exposes neutral read, mutation, transfer, and change primitives; sync direction, deletion propagation, local filesystem policy, and three-way reconciliation remain client concerns. + +### Baseline and changes + +1. A client obtains opaque start cursor `C0`. +2. It recursively enumerates a selected root using stable node and parent IDs with keyset pagination. +3. It replays changes after `C0` until caught up. +4. It persists the cursor after the final event it has applied. + +Enumeration and replay may contain duplicate observations but must not miss a committed mutation while the start cursor remains retained. No long-running PostgreSQL snapshot is held. + +The change feed is ordered by a monotonic sequence and consumers treat delivery as at least once. A page cursor represents the position after its final event. Each versioned event stores either an immutable resulting-node snapshot or a deletion tombstone, node and content revisions, subtree effect, origin information, and useful previous-location diagnostics. Cursor expiry returns `410 Gone` with stable code `change_cursor_expired` and requires rebaseline without discarding pending local work. + +Subtree-scoped credentials consume an authorization-safe projection of the global sequence. A move into the granted root is represented by current state, a move out is represented as a tombstone/removal, and unrelated events are omitted while the cursor advances. Recursive baseline and replay use the same root anchor and visibility rules, so a restricted client can maintain its authorized replica without learning metadata outside it. + +### Identity, revisions, and idempotency + +- Node revision increments for any meaningful mutation. Content revision increments only when committed file content changes. +- SHA-256 identifies committed content. Logical `content_modified_at` may preserve an authenticated client value; server `metadata_updated_at` records the latest mutation. Timestamps never decide conflicts. +- Authenticated client instances may bind to API tokens and are independently revocable. Application services receive a protocol-neutral mutation context for actor, credential, client, operation, idempotency, and request IDs. +- Idempotency records persist the authenticated scope, key, request fingerprint, operation, state, result reference, and expiry. Identical retries return the original result; a different fingerprint with the same key fails. + +### Capabilities and transfers + +`GET /api/v1/capabilities` reports protocol compatibility and supported behavior rather than requiring consumers to infer it from a software version. + +Upload creation may accept declared size, SHA-256, logical content mtime, target/precondition, and origin. After authorization, the server may avoid transfer or attach an already verified owner blob through the normal commit service. New content is always server-hashed and verified. + +Hash preflight is target-scoped and must not reveal whether matching content exists outside the credential's authorized view. The server may conceal whether internal deduplication occurred. + +## Consequences + +- Phase 1 owns the supporting schema and mutation context; Phase 2 owns the functional baseline, changes, capabilities, and conformance mirror. +- Web, importer, OnlyOffice, future desktop/rclone clients, and protocol gateways produce attributed events through the same application services. +- Client sync modes, SQLite schema, watcher behavior, local conflict copies, and provider adapters are deliberately deferred to the desktop-client design. diff --git a/docs/adr/0009-external-protocol-gateway-boundary.md b/docs/adr/0009-external-protocol-gateway-boundary.md new file mode 100644 index 0000000..2830104 --- /dev/null +++ b/docs/adr/0009-external-protocol-gateway-boundary.md @@ -0,0 +1,25 @@ +# ADR-0009: External protocol gateway boundary + +Status: Accepted + +## Context + +Existing backup and automation tools may benefit from WebDAV or rclone compatibility. Managed Drive storage is not a user-visible filesystem: PostgreSQL owns hierarchy and revisions while the array contains opaque immutable blobs. Direct protocol access to object paths would bypass authorization, commit intents, change events, trash, replacement recovery, and audit. + +## Decision + +- REST and application services remain canonical. +- A writable WebDAV gateway MAY be implemented post-v1 as a protocol translator. Every write calls normal application services, uses revision preconditions, creates normal audit/change records, and sends deletion to trash unless a credential explicitly permits permanent deletion. +- WebDAV uses independently revocable, action- and subtree-scoped credentials. Durable WebDAV lock tables are added with that module, not pre-created in the core schema. +- A native rclone backend MAY later call the REST API directly for richer hashes, modification times, resumable uploads, server-side move/copy, trash, and error semantics. Generic rclone-over-WebDAV and a native backend may coexist. +- Native rsync service, rsync staging ingestion, and direct SMB/SFTP access to managed object storage are not planned. Rsync may operate on a local folder maintained by a future desktop client, never on Drive's server object directory. +- No gateway may import storage adapters or mutate core repository tables directly (I16). + +Integration configuration and lock tables are introduced only when their module is implemented. Phase 1 adds only the shared foundations already required by REST clients: client identity, persistent idempotency, mutation attribution, and scoped credentials. + +## Consequences + +- Future compatibility does not force writable WebDAV or integration-specific tables into stable v1. +- Existing tools can gain interoperability later without weakening the immutable object model. +- A protocol gateway must provide streaming, conditional requests, bounded listings, authentication, audit, and revocation appropriate to that protocol. + diff --git a/docs/adr/README.md b/docs/adr/README.md index 59182f5..c76336e 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -11,4 +11,5 @@ ADRs are immutable after acceptance. Supersede an earlier decision with a new AD | 0005 | HTTP precondition and conflict status codes | Accepted | | 0006 | Stable-v1 and deferred product scope | Accepted | | 0007 | Open first-run setup | Accepted risk | - +| 0008 | Sync-capable server contract | Accepted | +| 0009 | External protocol gateway boundary | Accepted | diff --git a/docs/architecture.md b/docs/architecture.md index 72fc6f3..096fb4b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -16,9 +16,18 @@ Transport code handles protocol concerns only. Application services own transact No authoritative state lives only in memory. Upload sessions, commit intents, jobs, leases, and cursors are persisted in PostgreSQL. Committed blobs are immutable and become visible only after storage promotion and metadata commit. +## Sync-ready server contract + +The server exposes neutral file and change primitives, not client sync modes. A consumer obtains a start cursor, recursively enumerates stable node/parent IDs, then replays the durable change feed until caught up. Events carry resulting state or tombstones and are safe to process more than once. See ADR-0008. + +Node revision covers all meaningful mutations; content revision and SHA-256 distinguish content changes from rename, move, favorite, or trash metadata changes. Persistent idempotency and authenticated client attribution apply to every transport. + +## External protocols + +REST and application services are canonical. A future WebDAV gateway translates protocol requests into those services, and a native rclone backend may consume REST directly. Neither may touch managed object storage or repository tables. Rsync server access is not planned. See ADR-0009. + ## Frontend Each feature owns its routes, API hooks, components, state, and tests. Cross-feature access goes through a small public entrypoint. Server state stays in TanStack Query and local UI state remains close to the component or feature that owns it. Preview rendering is capability-driven and lazy-loaded. Adding a format must not require editing the file manager. - diff --git a/docs/threat-model.md b/docs/threat-model.md index 9b05de7..6d1d4f3 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -14,6 +14,7 @@ - Drive to the existing OnlyOffice Document Server. - Public-link visitors to explicitly shared resources. - Read-only importer mounts to managed Drive storage. +- Future WebDAV/rclone credentials to an explicitly granted Drive subtree and action set. ## Primary threats @@ -21,6 +22,7 @@ - Session theft, CSRF, credential guessing, token leakage, and proxy-address spoofing. - Path traversal, symlink following, MIME confusion, malicious previews, and archive expansion. - SSRF or forged OnlyOffice callbacks. +- Over-broad, leaked, or revoked integration credentials and protocol gateways that bypass application services. - Stale writes, duplicate requests, interrupted commits, and premature GC. - Secret disclosure through logs, images, configuration, backups, or repository history. @@ -31,6 +33,6 @@ - Non-root container, explicit mounts, private PostgreSQL network, and health endpoints. - Architectural boundaries prevent HTTP code from reaching database or managed storage directly. - Security headers, CSRF, authentication, rate limits, scoped tokens, and callback validation are implemented with their Phase 1 features. +- Phase 1 credentials are independently revocable and may be restricted by subtree and action; future gateways must use them and the canonical mutation path. Review this document whenever a new external integration, public endpoint, preview processor, or authentication mechanism is introduced. -