Kaleido/docs/runtime-verification.md
Elijah 430616330b
All checks were successful
Validate and publish Kaleido / android (push) Successful in 2m14s
Release 0.1.2 with navigation sounds and smoother Home scrolling
2026-10-24 10:18:59 -07:00

14 KiB
Raw Blame History

Device regression review - 2026-10-23

Target: the authorized onn at 192.168.50.195:5555, package tv.kaleido.app. Other applications and the launcher were not modified. Screenshots, traces, UI dumps and server-specific logs stay in ignored evidence/private/.

Browsing

  • Cold-start switching to Movies reproduced an empty page: tab loading cancelled server initialization before library discovery finished. Initialization now has its own job; section requests update their cache and visible results atomically. scripts/repro_runtime.py sections passes the same startup switch, TV loading and cached Movies return.
  • Home again has the large cinematic feature with Play/Resume, information and feature navigation. Movies has the compact focused-title preview with recent-release posters and recommendations. TV has wider next-episode thumbnails, newly added seasons and new episodes. A-Z remains a separate filtered poster grid.
  • All six scripts/verify_ui.py browse checks passed, including primary tab activation on focus, details/card focus restoration, library navigation and a live Plex PIN with cancellation preserving the connection.
  • Fixed card geometry and explicit row/column navigation remove horizontal vertical drift. Cached bitmap contrast reduces backdrop GPU work. API JSON parsing is off the main thread.
  • Profiling found substantial interpreted Compose/layout work immediately after ADB installation. Android reported status=verify, not profile-guided compilation. Installing the bundled baseline profile and compiling only Kaleido with speed-profile changed the measured browsing loop from p95 65 ms to 20 ms, median 14 ms, GPU p95 5 ms. Horizontal card bounds retained the same Y coordinate. This is a device stress measurement, not a guarantee of zero dropped frames. Baseline rules now also include app browsing paths.

For reproducible performance checks, install the release, send androidx.profileinstaller.action.INSTALL_PROFILE to Kaleido's ProfileInstallReceiver, compile its package with cmd package compile -f -m speed-profile tv.kaleido.app, then run scripts/repro_runtime.py browse. Do not benchmark debug builds or compare different compilation states. Ordinary sideloads receive profile-guided optimization through Android's background dex optimization; immediate first-run measurements can therefore be slower.

Playback and account connection

  • Six live Android integration tests passed: encrypted session persistence, library paging/filter/search/details/hubs, show/season/episode access, a real Plex PIN, generated JPEG previews/new seasons, and a server decision copying the supported HEVC test episode's video.
  • Next episode previously crashed with Session ID must be unique. Each controller now owns a unique Media3 session ID. The actual episode replacement loop passes and reaches the expected next title.
  • Plex HLS uses an absolute timeline. Removing the duplicate seek-start offset fixes resume/progress and credits-marker positions.
  • Natural episode completion reached post-play, waited five seconds and reached READY on the next episode. Credits skipping also opened season/related rows and its autoplay cancellation test passed.
  • Player tests exercised real video, scrubbing previews, audio/subtitle panels, speed changes and Direct stream selection. A separate targeted regression verifies that one Back hides controls, OK reveals them, playback auto-hides them and Back exits. Explicit preview-key handling fixes Back clearing focus before reaching the player handler.
  • Direct stream is the default preference. The client augments the existing Plex HEVC target instead of supplying competing transcode targets. The server can still choose conversion for media it cannot remux using the negotiated profile. Original-file and compatibility playback remain available.
  • The onn's ADB screen captures omit the tested HEVC picture even while Media3 reports rendered frames. The user confirmed that the actual TV displays the video. Do not treat that capture alone as a black-video failure or silently force transcoding.
  • Skip actions use an opaque contrast surface. The timeline draws buffered and played ranges separately and shows generated server previews or chapter artwork when available.
  • The release ZIP was checked for the actual private test token and .env: neither is present. The test connection is encrypted private device data. Settings exposes Plex account linking and server discovery.

Release compilation, unit tests and lint passed with zero lint errors (existing warnings remain). Temporary debugging log tags were removed.

The final release recheck after adding the app baseline rules passed again: median 15 ms, p95 27 ms, GPU p95 5 ms, stable row Y coordinates. Results remain dependent on device compilation state; some frames still miss a 60 Hz deadline under the burst test.

Rating artwork and startup review - 2026-10-24

  • Replaced the custom popcorn drawings with official red Hot / green Stale Popcornmeter vector paths. On-device screenshots show the red mark beside Andor's 89% and the green mark beside Matchbox's 49%; the existing IMDb score remains alongside them. Artwork fits a fixed square scaled with text, retaining the official aspect ratio and colors. See brand-assets.md for provenance and designation rules.
  • The 2.6-second opening assembles the six original aperture facets with light trails, a soft glow, a brief facet glint and a staggered wordmark reveal. Cover dismissal follows animation completion, then the bounded loading wait and crossfade. Android animation duration scale is honored by the animations; the device was reviewed at its existing 1.0 setting.
  • Built with Build.ps1 -Release -Test: optimized release, existing unit test and lint passed (zero errors, existing warnings/hints remain). Installed in place on the authorized onn with the existing encrypted session intact, then installed the baseline profile and compiled only Kaleido with speed-profile.
  • Recorded and inspected the actual cold-start sequence, Home and Movies/TV screenshots, and both rating states. verify_ui.py browse passed all six checks; repro_runtime.py sections passed startup tab switching, TV loading and cached Movies return. The harness now waits for the splash's accessibility description to disappear instead of counting wordmark text nodes.
  • Private evidence: ratings-home.png, final-movies-discover.png, kaleido-startup-review.mp4, and the trimmed kaleido-intro.mp4 in ignored evidence/private/. No playback checks or new frame-pacing benchmark were needed for these cosmetic changes. Left the optimized release on Home.

First published release and live updater test - 2026-10-24

Published Kaleido 0.1.1, version code 2, from commit c3917c6. Both the main validation workflow and matching tag publication workflow passed. The locally signed distribution retains the installed application's signing certificate. Anonymous requests to the release API, APK and checksum returned HTTP 200 without redirects; the release asset and immutable tagged repository APK matched the local SHA-256 (6f1d42ee5167dcb3743b7dafbecfed9942e496ee10b8b277ba2a3ba66c6407ba). The packaged APK was also checked for the private Plex token, .env and signing keys; none were embedded.

The onn stayed on version 0.1.0 / code 1 until the actual in-app installation. No ADB install of 0.1.1, downgrade, sign-out or app-data reset was used:

  1. In Settings, the existing app's manual check reached the public server. Before a Forgejo release existed, it discovered 0.1.1 using distribution/latest.json and its immutable tag URLs.
  2. After the tag workflow published the release, another manual check discovered the published release. Selecting Download & install 0.1.1 completed download, checksum, package, increasing-version and signing-certificate validation, then opened Android's source-permission screen.
  3. Enabled Kaleido only in Install unknown apps. Returning to Kaleido automatically opened Android's Do you want to update this app? confirmation. Selected Update, and Android reported App installed.
  4. Selected Open. Live Plex Home returned with the existing encrypted connection intact. Settings retained Direct stream, 1080p / 20 Mbps conversion quality and the 80% watched threshold.
  5. Android and Kaleido both reported 0.1.1 / code 2. The APK pulled from the installed package matched the published APK byte for byte. Another manual check reported Kaleido is up to date., with no redundant download action.
  6. Reinstalled the baseline profile and compiled only Kaleido with speed-profile; left the updated release on Home.

For future live checks, start with an installed lower version signed by the same key, publish a genuinely higher version/code, navigate to Settings > Software updates with the D-pad, and follow the sequence above. Never clear app data to manufacture an update scenario. Android's source permission is a one-time, per-app step; the package installer still requires explicit confirmation for each update. Wait for the TV Settings source list to populate before selecting Kaleido.

Private evidence remains in ignored evidence/private/: updater-results.json, published-release.json, updater-before.png, updater-fallback-discovery.png, updater-release-discovery.png, updater-source-allowed.png, updater-confirmation.png, updater-installed.png, updater-after.png and updater-final-home.png. This verifies manual discovery, the manifest fallback and a real release download/install. It does not independently test the six-hour automatic-check interval, permission denial/retry, or rejection of deliberately corrupted/wrong-signed APKs.

Navigation sounds and Home scroll performance - 2026-10-24

The specific hero-to-Continue Watching movement now has a dedicated check: scripts/profile_home.py <label>. It resets frame statistics for each Down transition, waits until it finishes, samples completed frames, then returns to the hero outside the measured interval. UI hierarchy dumps do not run during the measured animation. Its p95 limit is 33.34 ms; --cold --cycles 1 skips the warm-up scroll and --trace records app layout/rendering spans. Use an optimized release with the baseline profile installed and speed-profile compilation, and never run another device test alongside it.

The original profiled release failed across five repeated transitions: 59 frames, median 17.5 ms, p95 50.3 ms, maximum 60.5 ms, and 28.8% above 33.34 ms. A separate trace showed main-thread measure/layout passes of 40–51 ms, repeatedly composing and remembering newly exposed image cards. Render-thread spans were substantially shorter. This isolated lazy-row preparation/disposal as the primary bottleneck in this scenario.

Home now uses a bounded one-viewport-ahead / one-viewport-behind lazy-layout cache window. It prepares adjacent rows and retains the hero/nearby cards on the return path. Artwork resolution, card geometry, the cinematic hero and Movies/TV presentation are unchanged. The fix does not cache the entire catalog or change focus/scroll destinations.

With navigation sounds enabled and its hot paths included in the baseline profile, the final APK measured 66 frames across five repeated Down transitions: median 17.2 ms, p95 28.6 ms, maximum 34.8 ms, and one frame above 33.34 ms (1.5%). Its first transition after a cold process launch measured 13 frames, p95 26.9 ms and maximum 30.1 ms. These are device samples, not a guarantee of every frame meeting a 60 Hz deadline. A preliminary run where the profile-install broadcast targeted the wrong receiver was excluded from the matched comparison; the final preparation returned profile-install result 1 and package compilation Success.

Navigation sounds default to On under Settings > Interface. Shared focus targets call Android's native UI sound only after an actual focus change; startup focus, unchanged targets and disabled sounds are silent. The TV's global sound-effects setting remains authoritative and was not changed. verify_navigation_sounds.py reproduced the missing control on the original installation, then passed both Off and On persistence checks across cold process launches on the final build. Two unit tests cover initial/duplicate focus suppression and muting/re-enabling behavior.

For a quiet device with system effects already enabled, run python -c "import sys; sys.path.insert(0, 'scripts'); from verify_navigation_sounds import audio; audio()" using the configured Python runtime. It compares identical D-pad focus movements with the app preference Off/On: the final test recorded zero native audio mixer writes while muted and 141 while enabled. This confirms output through Android's audio pipeline; physical speaker listening was not performed. It leaves the preference On and returns to Home.

Private evidence includes home-baseline-results.json, home-trace-baseline-atrace.txt, home-complete-results.json, home-complete-cold-results.json, navigation-audio-results.json, navigation-audio-off-atrace.txt, navigation-audio-on-atrace.txt, and the On/Off setting screenshots. Build.ps1 -Release -Test passed with three unit tests and zero lint errors. These measurements were taken on the local optimized test build before versioning the changes as 0.1.2. The installed test build retains the existing session and was left on Home with navigation sounds On.

All six verify_ui.py browse checks passed with the sound setting and cache window, including focus restoration, separate Movies/TV layouts, A–Z grids and PIN cancellation preserving the session. The broader repro_runtime.py browse check initially failed at p95 42 ms (48 ms with tracing), with further card composition/measurement during rapid movement. Including the new sound modifier/controller in the app's baseline profile brought the final profiled release to median 15 ms / p95 36 ms, GPU p95 7 ms, and stable horizontal Y coordinates at 182 px. Some stress-test frames still miss 60 Hz. browse --trace now records this exact regression loop to browse-regression-atrace.txt for future diagnosis.