# Drive v2 — Final Architecture and Implementation Plan Status: approved baseline, updated 2026-07-16; Phase 0 complete. This document is the source of truth until superseded by ADRs in the repository. It is written for both human and AI implementers. Normative keywords MUST, MUST NOT, SHOULD, and MAY follow RFC 2119 meaning. Section numbers are stable and may be referenced from `AGENTS.md`, ADRs, and commit messages. --- ## 1. Product Definition and Deployment Context Drive v2 is a clean-replacement, personal, single-owner, self-hosted file platform. It is built in a new repository. The v1 application remains read-only as a behavioral reference; no schema, auth code, filesystem handlers, or UI components are copied from it. Deployment target (fixed, not generic): - Unraid 7.3.1, Docker. - Cache: two-drive btrfs mirror, exclusive share (FUSE bypassed), no mover. - Array: 6 data disks + 1 parity, 54 TB, accessed through `/mnt/user` via shfs/FUSE. Array shares CANNOT be exclusive; the implementation MUST NOT assume FUSE bypass for array paths. - Reverse proxy: Nginx Proxy Manager on the same host. - OnlyOffice Document Server: already running in a separate container on the same host. - Single owner. Multiple concurrent sessions, devices, and API tokens are supported. There are no other users. - Clients: desktop Chromium-based browsers (Brave), Android Chromium browsers, future desktop sync client. - Initial import corpus: a few hundred GB. Largest working files: 200-300 GB ZIP archives. ## 2. Non-Negotiable Invariants These are the promises the system makes. Focused automated tests (Section 13) encode them. AI implementers MUST NOT weaken, bypass, or modify these without a human-approved ADR. - I1. Committed metadata MUST never reference unavailable, incomplete, or unverified content. - I2. Managed content objects are immutable once committed. Nothing modifies a committed object in place. New content means a new object and a new revision. - I3. Every metadata mutation and its change-feed event commit in one PostgreSQL transaction. - I4. All sizes, offsets, ranges, and counters are 64-bit. Every transfer path streams; no code path may buffer an entire file in memory. - I5. Resources are identified by ID, never by path. Paths are derived presentation data. - I6. No authoritative state lives only in process memory: no in-memory-only uploads, jobs, locks, or queues. - I7. Only the storage adapter touches managed array paths. Only application services coordinate database plus storage mutations. Enforced by CI (Section 5). - I8. Errors are never silently discarded. - I9. Directory listing and search MUST never scan the array at request time. - I10. The server never applies silent last-write-wins. Failed HTTP revision preconditions return 412 with current server state; semantic conflicts return 409. - I11. In-flight (uncommitted) uploads MAY be lost on unclean shutdown. Committed data MUST NOT be. Crash recovery protects commits, not transfers. - I12. GC MUST NOT remove any object referenced by a live node, active export lease, replacement-recovery record, editor session, or incomplete commit intent. - I13. A start cursor plus complete recursive enumeration and subsequent change replay MUST be sufficient to reconstruct current server state without missing a committed mutation. - I14. Every mutation source MUST use the same application services, revision checks, audit path, and transactional change feed. - I15. Replaying a request with the same idempotency key and request fingerprint MUST NOT perform its mutation twice; reusing the key for different content MUST fail. - I16. External protocol gateways MUST NOT access managed object storage or mutate repository tables directly. - I17. Client-supplied hashes and modification times are comparison aids only; they MUST NOT bypass server verification, authorization, or revision preconditions. ## 3. Capacity Contract - No intentionally low application-level file-size limit. Default administrative upload ceiling: 1 TiB, raisable. - All transfer and hierarchy paths are designed for the stated file sizes and stream with bounded memory. Large-scale benchmarks and 300 GB qualification are deferred until representative product workflows exist and are not v1 release gates. - Unraid share minimum free space MUST be set to at least 400 GB (above the largest expected file class). - Free-space enforcement happens at upload-session creation (declared size vs. eligible-disk free space plus safety margin) and per accepted chunk. Finalization adds no bytes; a finalization-time space check is redundant and MUST NOT be the primary control. ## 4. Feature Scope ### 4.1 Stable v1 - First-run admin setup; password auth; session management; password recovery; TOTP 2FA; scoped API tokens. - Responsive file browser: breadcrumbs, grid/list, sorting, keyset pagination, virtualized rendering, keyboard navigation. - Folder create; file upload/download/rename/move/copy/trash/restore/permanent delete; multi-select; drag-and-drop and folder upload; resumable upload; cancellation; conflict handling; progress and speed. - Streamed range downloads, bulk downloads, streamed folder-archive downloads. - Filename and metadata search. - Pinned/favorite files. - Trash: 30-day default retention, configurable. - Replacement recovery: 7-day default window, configurable or disableable, with API and UI (Section 7.4). - Dashboards: storage, transfers, jobs, integrity findings, sessions/tokens, activity. (UI polish for these may be minimal at v1; backend data model is required.) - Scheduled integrity scrubbing and mark-and-sweep GC (Section 12.3). - pgBackRest backup with verified restore; blob manifest export (Section 12.4). - Required server-side bulk importer (Section 8.3). - OpenAPI-documented `/api/v1`; installable PWA (Section 10.4). - Previews: images (JPEG, PNG, GIF, WebP, SVG, BMP), ranged audio/video, PDF (PDF.js), bounded text/Markdown/CSV/JSON/logs, source code with highlighting, lazy viewport thumbnails. - Public link sharing: expiry, optional password, download limits, revocation, and abuse protection. - OnlyOffice viewing, editing, creation, and templates through the integration model in Section 11.1. - A complete sync-ready server contract: recursive baseline, durable changes, client attribution, capability negotiation, hashes, resumable transfer, and scoped credentials. The desktop client itself is deferred. - Light/dark themes. ### 4.2 Post-v1 modules and compatibility integrations - ZIP/ZIP64 and 7z archive viewing; durable zip/unzip jobs. - STL and 3MF 3D preview. - Calendar with ICS subscription feeds (Section 11.2). - Mirrored-folder desktop sync client. - Writable WebDAV compatibility gateway and an optional native rclone backend (ADR-0009). Rsync is not planned. No third-party plugin system, ever; extensions are native compiled modules. ## 5. Architecture ### 5.1 Backend Go, `net/http`, Chi, `pgx`, `sqlc`, versioned forward-only migrations. Layers: - Domain: identity, hierarchy, naming, revisions, conflicts, trash, replacement recovery, sessions, changes, job states. No HTTP, SQL, or filesystem imports. - Application: transactional use cases (create, upload finalize, rename, move, copy, trash, restore, import, preview preparation, editor-session commit). - Adapters: PostgreSQL repositories, array object storage, preview tools, OnlyOffice, calendar serialization, backup tooling, clock/ID providers. - Transport: thin handlers conforming to OpenAPI. Parse input, call one application service, serialize output. - Workers: PostgreSQL-backed jobs with leases, heartbeats, retries, cancellation, progress, `FOR UPDATE SKIP LOCKED`. ### 5.2 Boundary enforcement (CI, not prose) The layer rules above MUST be enforced mechanically via import-restriction linting (depguard or go-arch-lint in golangci-lint). A forbidden import is a build failure. Minimum rule set: - `transport` MUST NOT import `database/sql`, `pgx`, or `os` file APIs. - `domain` MUST NOT import transport, adapters, `net/http`, `pgx`, or filesystem packages. - Only the storage adapter package may reference managed storage path roots. - Repositories MUST NOT import transport. Rationale: AI implementers respect failing checks with near-perfect reliability and prose rules intermittently. Every architectural rule that can be a lint rule MUST be a lint rule. ### 5.3 Frontend React, TypeScript (strict), Vite, TanStack Router, TanStack Query. - TypeScript API client generated from OpenAPI; drift check in CI. - Feature modules: auth, files, uploads, search, trash, replacements, sharing, office, previews, jobs, settings; later calendar. - Server state in TanStack Query only; a small UI store for selection/view state; upload-session references persisted in IndexedDB. - Preview renderers lazy-loaded, selected by server-declared capability, never by a central extension switch. - Virtualized large lists. Accessible primitives, keyboard interaction, shared design tokens. - MUST NOT: global window event buses, multi-thousand-line components, localStorage for auth material. - NOT required: Storybook, full cross-browser matrix. ## 6. Storage Design ### 6.1 Unraid topology Cache (exclusive btrfs share, real filesystem semantics): - PostgreSQL data directory. - Application configuration and secrets. - Bounded preview cache (50 GiB default, LRU, configurable, evicted under pressure). Array (shfs user share, FUSE semantics assumed): - Managed objects and in-progress partials. - pgBackRest repository and blob manifests. The implementation MUST NOT claim or rely on cross-directory shfs rename atomicity. ### 6.2 Physical object layout (opaque IDs, same-directory promotion) Content is logically addressed by SHA-256 in PostgreSQL. Physical objects use opaque IDs: ```text /storage/.drive/ objects//.partial objects//.blob recovery/postgres/ ``` - Shard is derived from the object ID at creation time. - A partial and its completed object live in the same shard directory; promotion is a same-directory `.partial` → `.blob` rename. No cross-directory rename ever occurs after the final hash is known. - PostgreSQL maps SHA-256 → object ID. Deduplication is the database's unique SHA-256 constraint; if concurrent uploads race, one blob row wins and the redundant physical object is deleted only after the winning object is confirmed available. - Consequence and mitigation: the filesystem is no longer self-describing. The database is the only hash→object map. Therefore blob manifest export (Section 12.4) is a v1 REQUIREMENT with freshness monitoring, not a periodic nicety. - No `quarantine/` directory. Suspect content is marked unavailable in metadata and reported. ### 6.3 PostgreSQL on btrfs (ADR-001, decided) - PostgreSQL stays on the mirrored btrfs cache with copy-on-write ENABLED. `chattr +C` is rejected: it disables btrfs data checksums and defeats mirror self-heal. - PostgreSQL page checksums enabled at initdb. - Monitor DB size, WAL growth, fragmentation, cache-pool capacity. - Accepted cost: CoW fragmentation at single-user scale, covered by btrfs scrubs and pgBackRest. ### 6.4 Core tables `users`, `sessions`, `api_tokens`, `credential_root_grants`, `client_instances`, `idempotency_records`, `nodes`, `blobs`, `upload_sessions`, `commit_intents`, `replacement_recovery`, `changes`, `jobs`, `preview_assets`, `audit_events`, `settings`, `import_runs`, `import_items`, `shares`, `editor_sessions`, `export_leases`, `export_items`. Later module tables include archive indexes, calendar data, and WebDAV locks. Module tables MUST NOT add columns to core file tables. `nodes`: UUIDv7 id, owner, parent, name, case-folded name key, type, current blob id, size, MIME, monotonic node revision, monotonic content revision, logical content modification time, server metadata update time, created time, and trash-root time. Node revision changes for any meaningful mutation; content revision changes only when committed file content changes. SHA-256 and blob identity remain authoritative content identities. `changes`: monotonic sequence, event ID/type, schema version, node ID and revisions, subtree effect, immutable resulting-node snapshot or deletion tombstone, previous location/trash diagnostics where applicable, mutation origin, and commit time. `idempotency_records`: authenticated owner/token/client identity, key, request fingerprint, operation type, state, stable response reference, and expiry. Records persist across process restarts. `client_instances`: display name, type, platform, software/protocol versions, last seen time, and revocation time. API tokens MAY bind to a client instance and subtree/action grants. ## 7. File and Hierarchy Semantics ### 7.1 Portable naming - Unicode NFC normalization; case-folded uniqueness within a folder. - Reject: control chars, `/`, `\`, `.`, `..`, Windows reserved device names, trailing spaces/dots. - Deterministic conflict names: `name (conflict N).ext`, extension preserved. - Identical validation rules published in OpenAPI and enforced server-side. ### 7.2 Conflict policies - Default API behavior on name collision: `409 Conflict`. Never silently rename an API caller's request. - Explicit modes: `keep_both` (deterministic suffix), `overwrite` (requires passing revision precondition), `skip`. - Importer default: `keep_both`, with every rename recorded in the import report. ### 7.3 Large subtree operations - Move: mutates only the root node's parent and revision; emits ONE change event; clients derive descendant effects from hierarchy. - Trash: marks only the root as a trash root. Descendants become invisible via ancestor visibility, not row updates. O(1). - Restore: clears the root trash state, resolves root-name conflicts. O(1). - Permanent subtree delete: durable background job. Blob reclamation is separate scheduled GC. - Search returns indexed candidates then applies batched ancestor-visibility filtering. Direct node access checks the ancestor chain so stale IDs cannot reach descendants of trashed roots. Optimize this query when realistic data demonstrates a bottleneck. ### 7.4 Replacement recovery - On overwrite, the prior blob reference enters `replacement_recovery` with a 7-day default expiry (configurable, disableable). - GC cannot collect actively referenced recovery blobs (I12). - `/api/v1/replacements` and a "Recently replaced" UI: list timestamp/size/checksum, restore a prior blob as a NEW current revision, show storage consumed. - This is a short safety window, not version history. OnlyOffice saves flow through it too; document sizes make the cost negligible. ## 8. Transfers ### 8.1 Resumable uploads (TUS-compatible) - The Drive client defaults to 64 MiB sequential chunks. The server accepts bounded TUS chunks and enforces exact offset/length semantics without requiring one universal client chunk size. - Session records persist create/replace target, expected revision, declared size and optional SHA-256, logical content mtime, origin client/operation, conflict policy, accepted offset, state, and expiry. - Space check at session creation and per chunk (Section 3). - Survives graceful restarts and connection interruptions (the common case: closed tab, WiFi drop, container update). - Crash posture (I11): the incremental hash checkpoint is NOT required to survive unclean shutdown. After a crash, recovery MAY discard incomplete partials or re-hash them from disk; failing the upload is acceptable. Do not engineer durability for uncommitted transfers. ### 8.2 Commit protocol 1. Create persistent upload session; reserve target name. 2. Stream chunks to the `.partial` object. 3. Persist offset (and hash state, best-effort) per accepted chunk. 4. On completion: verify length, finalize SHA-256, fsync object and parent directory as supported. 5. Create commit intent. 6. Promote `.partial` → `.blob` (same directory). 7. Recheck target, revision, parent, and name preconditions. One PostgreSQL transaction then inserts-or-reuses the blob, creates/updates the node, increments node/content revisions as applicable, adds replacement recovery for overwrite, emits the attributed change event, and completes the intent. 8. Startup/periodic recovery workers finish or roll back incomplete intents (this protects I1/I11 and MUST run on every start). 9. Redundant deduplicated objects deleted only after the committed blob is confirmed available. ### 8.3 Required importer CLI inside the application image: ```text drive import --source /imports/data --destination / --conflict keep_both [--dry-run] ``` - Sources mounted read-only; sources inside managed storage are rejected; symlinks are not followed (reported); hardlinking into managed storage is PROHIBITED (a later source edit would mutate an "immutable" blob). - Copies through the same storage and commit services as uploads; streams SHA-256 during copy; deduplicates against existing blobs; preserves mtimes and empty folders where available. - Commits per file or bounded batch; persists run/item state; resumable and idempotent; pause/cancel supported. - Reports: copied, deduplicated, renamed, skipped, invalid, failed, changed-during-import. - Dry-run: fast scan (counts, bytes, invalid names, symlinks, case collisions); full analysis optionally hashes sources to estimate dedup. - Sized for the real corpus: a few hundred GB initial ingest. No multi-batch migration ceremony is required; run it, read the report. The importer is an ingestion tool, not a database backup substitute. ### 8.4 Small files TUS-per-file overhead is accepted for v1 and measured, not hidden. Post-v1 candidates: batch small-file endpoint, upload-archive-and-unpack with strict limits, desktop-client batching. ## 9. API and Sync Contract OpenAPI 3.1 is the public source of truth and lives in the repository. Stable-v1 endpoint families: `/api/v1/setup`, `/sessions`, `/api-tokens`, `/client-instances`, `/capabilities`, `/nodes`, `/nodes/{id}`, `/nodes/{id}/children`, `/nodes/{id}/descendants`, `/nodes/{id}/content`, `/uploads`, `/search`, `/trash`, `/replacements`, `/changes`, `/jobs`, `/previews`, `/shares`, and `/office`; `/health/live`, `/health/ready`, and `/metrics`. Calendar endpoints are added only when the deferred Calendar module is implemented. Rules: - Opaque keyset cursors; no offset pagination on large collections. - RFC 9457 problem responses with stable machine-readable error codes. - `ETag` + `If-Match` or `expected_revision` on mutations. Failed representation preconditions return `412`; name/hierarchy/domain conflicts return `409` (I10 and ADR-0005). - `Idempotency-Key` on create, finalize, copy, move, import, restore, trash/delete, and other retryable mutations. The server persists the request fingerprint and original result (I15). - Authentication establishes a protocol-neutral mutation context: actor, credential, validated client instance, operation ID, idempotency key, and request ID. REST, importer, OnlyOffice, and future gateways pass this same context to application services. - Standard `Range`/`Content-Range`/`Content-Disposition`, cache validators, and RFC 9530 `Content-Digest`/`Repr-Digest` where applicable. SHA-256 is also present in JSON node metadata. - API tokens can be restricted by action and root node. External credentials default to trash rather than permanent deletion and do not gain access outside their granted subtree. - Optional declared SHA-256 and size are accepted during upload creation. After target authorization and precondition checks, the server MAY perform a no-op or attach an already verified owner blob through the normal commit service; newly transferred bytes are always server-verified (I17). - Folder move/trash/delete events apply to the subtree rooted at the referenced ID (one event, Section 7.3). ### 9.1 Capability negotiation `GET /api/v1/capabilities` reports API and sync protocol versions, minimum compatible client protocol, hash algorithms, naming rules, timestamp precision, change retention, administrative upload ceiling, and supported features such as recursive baseline, change cursors, resumable upload, server-side move/copy, logical content mtimes, trash, replacement recovery, and root-scoped credentials. Clients MUST use capabilities instead of inferring behavior from the server software version. ### 9.2 Consistent baseline protocol 1. Obtain opaque cursor `C0` from `GET /api/v1/changes/start-cursor`. 2. Enumerate the selected root with keyset-paginated `GET /api/v1/nodes/{rootId}/descendants`, returning stable node/parent IDs and requested metadata fields. 3. Replay `/api/v1/changes?cursor=C0` until caught up. 4. Persist the cursor returned after the last processed event. Concurrent mutations may be observed both in enumeration and replay, so duplicate observations are expected. This sequence MUST not miss committed mutations while `C0` remains within retention (I13). It does not hold a long-running PostgreSQL snapshot. ### 9.3 Change delivery contract - `/changes` is the durable authority; SSE only signals that polling may find work. - Events are ordered by monotonic sequence and consumers treat delivery as at least once. Page cursors identify the position after the final returned event. - Each event carries a versioned resulting-node snapshot or deletion tombstone, node/content revisions, subtree effect, origin client/operation, and useful previous-location diagnostics. Resulting state is authoritative. - Processing by node ID plus revision is idempotent; consumers tolerate duplicate events, already-applied revisions, repeated subtree effects, and restart between side effect and cursor commit. - A subtree-scoped credential receives an authorization-safe projection of the global sequence. Moves into its root appear as current snapshots; moves out appear as tombstones/removals; unrelated events are omitted while the cursor still advances. Baseline enumeration and change replay use the same root anchor and visibility rules. - Change/tombstone retention is 365 days. An expired cursor returns `410 Gone` with stable code `change_cursor_expired`; consumers preserve pending local work, establish a new baseline, and reconcile rather than discarding it. - Every mutation source emits through this same transactional mechanism (I3 and I14). ### 9.4 Sync conformance mirror (permanent tool) Phase 2 builds a one-way download mirror: baseline the tree, poll `/changes`, download content, apply rename/move/trash/delete, persist cursor, survive restart, handle tombstones, folder moves, forced rebaseline, and verify checksums. Its focused contract tests cover mutation during enumeration, duplicate events, interrupted rebaseline, cursor expiry with pending local state, restart before cursor commit, queued descendant work during folder moves, and originating-client acknowledgement. It stays in the repository as the regression harness. A future bidirectional client consumes the contract it proves. ## 10. Capability Providers The server returns per-node capabilities; the frontend maps manifest types to lazy renderers. Each provider defines recognition, resource limits, preparation mode, manifest schema, caching, cleanup, and security validation. Adding a preview means adding a provider + manifest + renderer + fixtures, never editing the file manager. ### 10.1 Core v1 providers direct-image, ranged-media, pdf, bounded-text, source-code. ### 10.2 Thumbnails - Lazy generation on first need; batched prefetch limited to the visible/near-visible viewport. - Deduplicated by source blob hash + variant; served from the bounded LRU cache. - Eager whole-library generation at ingest is REJECTED. Never enqueue thumbnails for the entire imported corpus. ### 10.3 Archives (post-v1) - ZIP/ZIP64: ranged central-directory reads without extraction; async indexing; paginated entry browsing; individual-entry download. Tested against real 300 GB ZIP64 archives and high entry counts. - 7z: listing and whole-archive operations. Individual-entry extraction only when bounded; solid archives may require decompressing preceding block data, so the provider MUST estimate cost, warn, and reject beyond configured CPU/temp-space/ratio limits. - Protections: traversal, entry-count, total-output, expansion-ratio, nesting limits; cancellation and cleanup; no synchronous extraction in an HTTP handler. ### 10.4 PWA - Installable manifest; service worker caches ONLY the app shell and safe static assets; MUST NOT cache authenticated file bytes, downloads, or previews. - Web Share Target: feature-detected Android/Chromium enhancement. Shared files enter a recoverable pending upload feeding the normal streaming pipeline (never buffered in service-worker memory); ordinary upload is the fallback. - Offline browsing/content is out of scope for v1. ## 11. OnlyOffice and Calendar ### 11.1 OnlyOffice integration model (with existing Document Server container) Operating model, stated explicitly because it is commonly misunderstood: OnlyOffice Document Server holds the working copy internally during an editing session. Keystroke autosave happens INSIDE Document Server, not against Drive storage. Drive receives content only at discrete callback events (periodic forcesave if enabled; final save on session close). Therefore continuous editing is fully compatible with immutable blobs (I2): every callback save is a normal staged commit producing a new blob and revision. Requirements: - Editor config document key MUST incorporate node ID + revision, so a committed revision invalidates Document Server's cached session key and stale content is never served. - Callbacks MUST be JWT-authenticated with a shared secret; Document Server host allowlisted; SSRF protections on all outbound fetches. - Document Server fetches content through short-lived scoped content URLs tied to the editor session, not user session cookies. - Callback commits carry `expected_revision`; on mismatch (file replaced mid-session) the save lands as a conflict copy, never a clobber (I10). - Active editor sessions pin their source blobs against GC (I12). - Saves flow through staging + commit protocol (Section 8.2). Forcesave-generated revisions pass through replacement recovery normally; office-document sizes make this cheap. No direct filesystem access by Document Server, ever. - Blank documents and templates create nodes through the normal node service. ### 11.2 Calendar (isolated module, own tables, `/api/v1/calendar`) - Calendars, events, all-day and timed events, IANA time zones, recurrence with exceptions and moved instances, reminders, attachments, ICS import/export. - Recurrence MUST use a maintained RFC 5545 library (selected by ADR with conformance fixtures covering DST boundaries, EXDATE/RDATE, moved instances). Hand-rolled RRULE evaluation is PROHIBITED. - Read-only ICS subscription feeds: `/api/v1/calendar/feeds/.ics`. Tokens are high-entropy, stored hashed, scoped to one calendar, revocable, rotatable; UI warns that URL possession grants read access. Writes remain via UI/API. No CalDAV. - PWA calendar views. Reminder definitions may be stored, but notification delivery is deferred from the initial Calendar release. ## 12. Security, Reliability, Operations ### 12.1 Security - Argon2id with versioned parameters; server-side revocable sessions in `HttpOnly`/`Secure`/`SameSite` cookies; CSRF protection on cookie-authenticated mutations; hashed action- and subtree-scoped API tokens; client-instance binding and independent revocation; no long-lived browser JWTs. - Trusted-proxy configuration; rate limiting on validated client IP; strict CSP and security headers; MIME-sniff protection; symlink rejection; archive protections (10.3); OnlyOffice callback auth and SSRF protection (11.1). - Application-level encryption at rest: not included (use Unraid/disk encryption if desired), preserving dedup and recovery practicality. ### 12.2 Shutdown and power - UPS with Unraid-managed safe shutdown strongly recommended and documented. - On SIGTERM: stop new sessions and finalizations, let in-flight chunk writes reach a recoverable boundary, release job leases, flush logs, exit within the container stop timeout. - Deployment docs MUST specify raising the Docker container stop timeout above Unraid's 10-second default and aligning Unraid shutdown timers, otherwise the SIGTERM grace period exists only on paper. - Unclean restarts ALWAYS run commit-intent and upload-session recovery. Losing in-flight uploads on unclean shutdown is acceptable by design (I11). ### 12.3 Scheduled array work Scrub and GC MUST support: configurable time windows, bandwidth/IOPS limits, pause/resume, checkpoints at shard boundaries (no full restarts), automatic throttle/pause during interactive transfers, and (where feasible) bounded concurrently-active disks. Integrity failures mark blobs unavailable and create actionable findings. ### 12.4 Backup and recovery - pgBackRest with WAL archiving to the array; documented retention; backup failures surfaced in health and admin UI; periodic restore test into an isolated PostgreSQL instance. - Blob manifest export (object ID ↔ SHA-256 ↔ size ↔ node path snapshot) written to the array on a schedule. Manifest freshness is monitored and ALERTED like backup freshness, because with opaque object IDs the database is the only map (Section 6.2). - Documentation states plainly: parity, the cache mirror, and on-server pgBackRest are availability layers on one physical machine, not backups. Independent/off-server backup is recommended for metadata and irreplaceable content. ### 12.5 Observability Structured JSON logs (request/user/node/upload/job IDs). Prometheus metrics: request latency/errors, transfer throughput, active/interrupted sessions, job depth/retries/failures, import progress, DB/WAL/preview-cache growth, array and cache capacity, integrity findings, backup and manifest freshness, disk-job activity, replacement-recovery storage. ## 13. Testing Policy (Recalibrated) Testing is proportionate to a private single-owner application. Early work favors normal unit tests and focused integration tests over synthetic scale, endurance, or exhaustive crash matrices. - Go and TypeScript unit tests cover naming, hierarchy, revisions, idempotency, change ordering, deduplication, conflicts, replacement recovery, and UI behavior. - Focused PostgreSQL/storage integration tests cover the commit protocol, mutation-plus-change transaction, retry behavior, and core API operations. - The sync conformance mirror permanently covers the baseline and delivery guarantees in Section 9. It uses ordinary fixtures, not massive generated trees. - A limited Playwright suite covers setup, login/2FA, upload, CRUD, conflicts, sharing, OnlyOffice, previews, trash, recovery, and session expiry. Include an accessibility smoke test for login and the primary file workflow. - Security tests cover traversal, symlinks, malformed ranges, CSRF, token/grant revocation, proxy IP handling, SSRF, unauthorized access, and gateway boundary enforcement. - Import tests cover deduplication, collisions, invalid names, changed sources, read-only sources, empty folders, and a normal restart. - Before production adoption, perform a clean Unraid install, normal upgrade/restart, representative large-file transfer, importer run, backup, and one restore. Deferred until real workloads justify them: 10-million-node benchmarks, 500,000-node subtrees, 300 GB certification, PostgreSQL/btrfs benchmarks, repeated process-kill matrices, endurance suites, and full browser coverage. Add targeted stress or recovery tests when actual profiling or failures identify a risk. Hash-checkpoint durability for uncommitted uploads remains explicitly unnecessary (I11). ## 14. AI Implementation Policy Most code will be written by AI agents (multiple models over time). The repository, not chat history, is the agents' world. - This plan's contracts live IN the repository: OpenAPI spec, SQL schema + sqlc queries, ADRs, and per-module docs. `AGENTS.md` references them by path and section. - Contract-first is anti-hallucination infrastructure: agents cannot invent endpoints (generated client + drift check) or columns (sqlc compile failure). The human owns the OpenAPI spec, schema, and sqlc queries as reviewed artifacts; agents implement between them. Own the interfaces, delegate the implementations. - Architecture boundaries are lint failures, not guidelines (Section 5.2). - The invariant test suite (encoding Section 2) is AGENT-IMMUTABLE: it lives in a dedicated directory that agents may not modify without explicit human approval; any diff touching it requires human review. Making a red test green by editing the test is the canonical failure mode this prevents. - Every AI change: small, independently testable, reviewed against the invariant it implements. Dependency APIs verified against primary documentation; no invented methods. No new dependencies without approval. No unrelated "misc fixes" in a change. - Data mutations ship with focused success, conflict/failure, and idempotency coverage appropriate to their risk. Add restart/cleanup cases where durable work or cross-system commits require them. - Architectural changes require an ADR before code. - A feature is complete when its expected errors, authorization, cancellation where applicable, observability, and recovery behavior are defined and tested proportionately. - `AGENTS.md` stays short and operational: build/test/lint commands, boundary rules, Section 2 invariants, forbidden patterns, and ADR pointers. Long instruction files degrade agent compliance. ## 15. Implementation Phases ### Phase 0 — Repository and development foundation (complete) - Repository conventions, `AGENTS.md`, ADR process, threat model, dependency policy, Go/React module boundaries, OpenAPI/sqlc skeletons, Compose environment, Forgejo CI, and versioned container build. - Minimal health/setup-status server and frontend shell with boundary, unit, contract, and image smoke checks. ### Phase 1 — Core domain, persistence, authentication, and storage - Migrations/repositories for hierarchy, node/content revisions and timestamps, replacement recovery, attributed changes, persistent idempotency, client instances, scoped credentials, jobs, settings, and audit events. - Owner setup, sessions, recovery codes, TOTP, API tokens, CLI recovery, CSRF, trusted proxy, health, metrics, and structured logs. - Storage objects, commit intents, promotion/deduplication, recovery sweep, resource leases, reference tracking, and GC eligibility. - Exit when focused unit/integration tests demonstrate that committed metadata cannot reference incomplete content and mutations cannot detach from their change event. ### Phase 2 — Transfers, importer, and sync-ready server contract - TUS uploads with pause/resume/cancel, offset reconciliation, optional hash preflight, explicit commit, range/bulk/folder downloads, and logical content mtime. - Start cursor, recursive enumeration, complete change polling, SSE hints, capability negotiation, tombstones, and permanent conformance mirror per Section 9. - Importer dry-run, resumable run state, conflicts, deduplication, reports, and representative real-corpus ingestion. - Search, favorites, trash, replacement recovery, backup tooling, manifest export, and practical integrity checks. - Exit when ordinary interruption/retry scenarios converge without duplicate mutation or missed change. ### Phase 3 — File operations and durable maintenance - Complete create, rename, move, copy, trash, restore, permanent delete, replacement recovery, favorites, search, and batch operations. - Large subtree copies/deletes, export snapshots, scrub, and GC as persistent checkpointed jobs with progress/cancellation and safe cleanup. - Exit when all normal file-management operations and conflicts are functional through the API. ### Phase 4 — Core web application - Setup/authentication, responsive file navigation, list/grid views, keyset pagination, virtualization, keyboard navigation, selection, uploads, conflict dialogs, search, favorites, trash, replacements, and operational dashboards. - Maintain feature import boundaries, component-size limits, accessibility, responsive Brave/Android layouts, and light/dark themes. - Exit when day-to-day file management works without relying on previews or office integration. ### Phase 5 — Stable-v1 sharing and OnlyOffice - Public file/folder links with passwords, expiry, limits, revocation, throttling, audit, and snapshot-backed folder downloads. - OnlyOffice scoped sessions, revision-bearing keys, JWT callbacks, conflict-copy saves, GC pinning, blank documents, and templates. ### Phase 6 — Stable-v1 previews and PWA - Capability-driven image/media/PDF/text/source previews, viewport-driven thumbnails, bounded cache, resource/security limits, installable PWA, and Android Share Target. ### Phase 7 — Practical hardening and stable-v1 release - Standard automated suite, focused security review, container scan, clean Unraid install, upgrade/restart, representative large-file workflow, importer run, backup, and one restore. - Publish Unraid, proxy, setup order, OnlyOffice, shutdown, backup/restore, importer, capacity, and troubleshooting documentation. ### Deferred modules - Desktop mirror client: standalone three-way reconciliation engine with capability-aware remote/local adapters, local SQLite state, watcher-as-hint plus scans, whole-file resumable transfers, deterministic conflict copies, and a thin Wails UI. On-demand placeholder/stream mode is a separate OS-specific product. - Calendar: isolated schema/API, maintained RFC 5545 recurrence library, ICS feeds, PWA views, and no initial reminder delivery. - ZIP/ZIP64/7z and STL/3MF providers with bounded durable work. - Writable WebDAV gateway and optional native rclone backend per ADR-0009. Neither is the official live-sync engine. ## 16. Fixed Assumptions and Decision Log Seeds - Single owner; multiple sessions/devices/tokens. No migration of v1 metadata or URLs; content ingested via the verified-copy importer. - Drive exclusively owns managed objects after import; external writes into managed storage unsupported; hardlinks into managed storage prohibited. - Latest content is canonical; 7-day replacement window; 30-day trash. - PostgreSQL, config, previews on the exclusive btrfs cache; objects and recovery data on the array share via shfs. - REST and application services are canonical. Future clients and native rclone use REST; protocol gateways translate into the same services and never touch managed objects directly. - Rsync support and direct SMB/SFTP access to managed object storage are not planned. Writable WebDAV is an optional post-v1 gateway. No plugin system. No CalDAV in the planned Calendar module. - The accepted ADR index in `docs/adr/README.md` is authoritative. New architectural decisions require a new ADR rather than edits to accepted ADR history.