Kaleido-Desktop/docs/architecture.md
Elijah 5b6c063ff1 Use nlmeans and FSRCNNX 16 for the Quality upscaling tier
ArtCNN is tuned for animation and looked unnatural on live action, and
its first Direct3D compile took about a minute per model. Quality now
denoises with AN3223's nlmeans (general-purpose profile, any size) and
upscales with FSRCNNX 16-0-4-1, trained on general imagery, at any
enlargement. Both compile in seconds; 1080p in a maximized 1440p window
measured 3.2 ms per frame.

The first-compile notice now covers any shader set not yet compiled on
the machine but only shows after three seconds, and a set with nothing
to verify at the current size clears it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 19:51:47 -07:00

5.3 KiB

Architecture

Kaleido Desktop is a Tauri 2 application: a Rust core hosts the system web view (WebView2 on Windows, WKWebView on macOS, WebKitGTK on Linux), which runs the React/TypeScript interface. Video is decoded and rendered by libmpv and composed beneath the transparent web view, so the player's controls are ordinary interface elements drawn over the video — the same arrangement Plex HTPC and Jellyfin Media Player use.

src/                      Shared interface (all platforms)
  app/                    Session, router, settings, queries, actions, shell, updater
  plex/                   Plex client: account (plex.tv), library API, admin, playback, parsing
  player/                 Playback controller (store.ts), mpv bridge, player overlay
  downloads/              Download catalog and actions
  screens/                Home, Movies/TV (Discover/Library/Categories), title pages,
                          search, people, downloads, settings, dialogs
  ui/                     Design system: primitives, menus/dialogs/toasts, cards, hero
  styles/                 Tokens and styles (base, ui, app, screens, player)
  platform/native.ts      The only module that talks to the host (Tauri or browser)
src-tauri/                Shared Rust core
  src/mpv.rs              Dynamically loaded libmpv binding (node ↔ JSON)
  src/player.rs           One long-lived mpv instance, event stream to the interface
  src/downloads.rs        Resumable downloads with a JSON catalog
  src/system.rs           Credentials (OS keychain), paths, disk space, file saving
  src/platform/           Everything OS-specific (see below)
  tauri.conf.json         Shared configuration (+ tauri.<os>.conf.json overrides)
vendor/mpv/<os>/          libmpv per platform (fetched by scripts, ignored by git)
scripts/                  build, libmpv fetch, update-feed generation
updates/latest.json       Signed update feed for every platform

Platform boundary

src-tauri/src/platform/mod.rs selects one of windows.rs, macos.rs or linux.rs. Each exposes the same items:

Item Windows implementation
VideoSurface mpv renders with d3d11-output-mode=composition; its swapchain (display-swapchain) is set as the content of a DirectComposition visual targeted at the window with topmost = false, i.e. behind the web view's child windows. For the miniplayer the visual is offset and mpv's composition size shrunk to a corner rectangle (player_rect).
MPV_LIBRARIES libmpv-2.dll
hold_awake SetThreadExecutionState while video plays

macOS and Linux currently provide stubs for VideoSurface (browsing, downloads, settings and updates work; playback reports that it is unavailable). To add playback there, render through mpv's render API into a layer beneath the web view and implement the same methods (set_rect places the video in part of the window for the miniplayer). Per-OS bundling (libmpv, targets, dependencies) lives in tauri.macos.conf.json and tauri.linux.conf.json. Credentials use the platform's keychain through the keyring crate with per-OS features in Cargo.toml.

On Windows the main window is created transparent and the page background turns transparent only while video is visible (html.video-active), so browsing is opaque.

Interface

  • State: zustand stores for the session (app/session.ts), navigation history (app/router.ts, with scroll and per-screen state restored on Back), preferences (app/settings.ts, persisted) and the player (player/store.ts). Server data goes through TanStack Query; every key starts with the server's machine identifier.
  • Plex HTTP goes through the host's native HTTP client (@tauri-apps/plugin-http), avoiding CORS and local-network restrictions. Artwork uses Plex's photo transcoder, sized for the display's pixel ratio (PNG for transparent clear logos).
  • Library sync: the section list is polled every 20 seconds; a change refreshes browsing caches.
  • Playback (player/store.ts): Original quality plays the part URL directly (multi part titles are joined with an mpv EDL); other qualities request Plex HLS with timestamps kept absolute. If direct play fails, playback retries through Plex once. Plex streams are matched to mpv tracks by stream index; external subtitles are added on demand. Timeline reports every 10 seconds make the session appear in Plex, and the title is marked watched at the configured percentage. The player's back arrow (or Esc) shrinks playback into a miniplayer: the video is drawn in a corner rectangle and the interface stays visible around it, with a matching hole masked out of the page (html.player-mini).
  • Upscaling (player/enhance.ts): tiers set mpv's scalers and debanding and load shaders bundled from src-tauri/shaders/ (FSRCNNX, nlmeans, KrigBilateral, AMD CAS); each shader's WHEN condition decides whether it runs for the current video and size. The chosen tier and the running one are tracked separately: shader errors in mpv's log or missing network passes in vo-passes drop to the tier below. Compiled shaders are cached in the app cache folder (gpu-shader-cache-dir), so a slow first Direct3D compile happens once per shader set, mpv version and GPU driver.
  • Updates: app/updater.ts checks the signed feed on start; installs need the user's confirmation and then relaunch.