Luma/docs/VALIDATION.md

139 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Device validation
Validated on the user’s onn. 4K Plus Streaming device over ADB, October 17, 2026 (America/Phoenix). Android 14 / API 34, approximately 1.88 GiB physical RAM, 1920 × 1080 render override on a 4K display.
## Functional checks
- Native app discovery, D-pad focus across rows, OK launch, long-press management menu, and settings navigation.
- Artwork only, Dynamic, and Minimal banner modes; original spacious layout restored as the default. Focused-only labels and hidden app-section text are the defaults.
- Smooth focus scrolling across all app rows, including the All apps footer, and back to the first row.
- Plex connection, bounded rows, artwork, encrypted token storage, and disconnect verified against a temporary local fixture. Fixture credentials were removed. A real Plex account/server has not been connected; actual Plex authorization and title playback remain unverified.
- Art screensaver: all three bundled paintings decoded and displayed, sampled for nonblank pixels, with pure-black side mattes. Captions have a dark gradient and text shadows. Paintings retain their full aspect ratio in Auto/Fit.
- Scenic photo preview, single-button wake, and real one-minute idle activation passed. Idle timeout was restored to ten minutes, photo duration to thirty seconds, and mode to Scenic.
- NPS bird video downloaded to private device storage (about 40 MB), played locally in a loop, and stopped on leaving Luma. The local file remains available in Nature video mode; no ongoing stream/download is active.
- Home routing: Android reports Luma as HOME role holder but the OEM resolver still chooses Google Home. The enabled Luma accessibility service routes Home to Luma. Repeated Home presses, return from Android settings and SmartTube, and screensaver dismissal were checked.
- Projectivy fallback stays open when explicitly selected through its app tile or Settings. Back stays on Luma; from lower rows it returns to the first row. Home returns to Luma. Projectivy remains installed and enabled; only its competing “Override current launcher” preference is off. Its accessibility service and Button Mapper’s service are retained.
- Luma’s override Off control removes only its own accessibility service. Re-enabling through Android’s actual Accessibility settings was verified. The service is system-bound in a separate `tv.luma.launcher:home` process. No view-content retrieval capability is requested.
## Wake regression — October 18, 2026
The original blank state was captured before sending input or restarting Luma. Android reported the existing activity as resumed; tiles were displaying their text placeholders and the alpine bitmap was absent. The recorded wake sequence included start/resume, a brief pause, and then resume without another start. Image work scheduled only from `onStart()` could be skipped or discarded while paused, with no retry when the activity resumed.
Version 1.0.1 moves restoration to `onResume()`, retries interrupted app discovery, and reloads missing wallpaper, app icons, and optional media artwork. Pausing invalidates stale asynchronous results; stopping still releases image references and caches. A pause-only resume retains existing tile views and decoded app images. This follows Android's documented [pause/resume lifecycle](https://developer.android.com/guide/components/activities/activity-lifecycle).
A separate same-signature instrumentation APK reproduces the interrupted-load sequence deterministically. The installed 1.0.0 build failed with missing artwork; the fixed build passed the same case, recovery of a partially rendered grid whose icon jobs were interrupted, pause-only memory trimming, reuse of existing loaded tiles/bitmaps, and five complete stop/start cycles. No preferences are rewritten by these tests. The temporary diagnostic APK was removed afterward.
One real sleep/wake cycle also passed screenshot checks for the alpine wallpaper and Plex banner without pressing Home or restarting the launcher. A subsequent automated wake was interrupted by HDMI standby (`mLastSleepReason=hdmi`), and One Touch Play timed out; no CEC settings were changed. The user turned the TV on again for the final screen check. The final screenshot shows the alpine wallpaper and app banners restored, with the user’s five-column layout preserved. Both the launcher and Home helper recorded zero CPU ticks over a ten-second background sample. Home returned correctly from Android settings and restored the UI after its cached process was explicitly reclaimed. The Home helper was re-bound after instrumentation cleanup, preserving the original enabled services. This is not an overnight endurance test.
## October 17 performance baseline
Final device sample with the Home override enabled:
| Measurement | Result |
| --- | --- |
| Launcher UI foreground PSS | 95,566 KiB (93.3 MiB) |
| Launcher UI cached/background PSS | 66,635 KiB (65.1 MiB), reclaimable |
| Separate Home router PSS | 6,348 KiB (6.2 MiB) |
| Launcher background CPU | 0 CPU ticks over 10 seconds |
| Home router idle CPU | 0 CPU ticks over 10 seconds |
| Focus navigation rendering | 84 frames, 4 janky (4.76%); 95th percentile 24 ms |
The separate Home router avoids retaining the UI image heap. Android can reclaim the UI while the router stays bound; Home then creates the UI again. This is not a guarantee of perfect 60 fps.
Raw measurements are recorded in `../output/luma-*-memory.txt`, `../output/luma-background-cpu.txt`, and `../output/luma-focus-frames.txt`. PSS includes framework/graphics allocations and fluctuates with system state. Short CPU samples indicate idle behavior, not a guarantee covering all usage.
## Scope
No launcher application was disabled or uninstalled. Screensaver selection at the Android system level was not changed. Standard Android Dream settings are unavailable on this TV, so the customizable idle screensaver runs while Luma is foreground. No real Plex credentials were extracted or used. The UI-reclaim test explicitly killed the cached UI process while leaving the Home router bound, then verified that Home restored Luma. No reboot test was performed.
Build validation uses aapt2, javac, D8, zipalign, and apksigner signature verification. There are no third-party runtime libraries.
Final APK (1.1.8): 2,753,622 bytes. SHA-256: `9f4fecf5162c897e9614b9fe4b3af37989bd10d3959d9278f1314e4e0630b80b`.
## Updater regression — October 18, 2026
The same-signature updater test ran against the installed 1.1.0 bootstrap and a newer signed 1.1.1 APK. It passed persisted daily network scheduling, Off cancellation, branch-to-commit pinning, a real APK download and certificate/package/version/checksum verification, and private provider read access. It rejected an incorrect checksum, another signing key, a different package, a mismatched version code, oversized metadata declarations, and public HTTP sources. The provider rejected write requests and unknown paths. The current version was not offered as an update. Test preferences were restored afterward.
The production default is the public HTTPS Forgejo feed in `output/update.json`, generated with the APK by `scripts/build.sh`. It requires no credentials on the TV. Native installer and unknown-source permission activities resolve on the onn. Android 14 device.
## Production self-update verification
A forced scheduled check against the real public HTTPS repository found a newer version while Android settings stayed foreground. Android recorded the Luma update notification, and Home showed the Update button. Downloading and verification used the commit-pinned APK from Git, not ADB. The first-use Android install-source permission screen was exercised.
Testing exposed a device-specific consequence of legacy APK/file installation: Android restricted settings reset the accessibility service list. The final flow uses a PackageInstaller session for a publisher-verified self-update (`PACKAGE_SOURCE_OTHER`), always requiring user confirmation. It does not claim to be an app store or request silent installation. Unlike a general APK file installer, Luma accepts only its own package signed by the installed key.
The final real upgrade, **1.1.4 → 1.1.5**, was downloaded from production Git and approved with the TV remote. The complete enabled-accessibility-service list matched before and after, including Luma Home, Projectivy, and Button Mapper. Restricted-settings authorization remained allowed, and the TV returned directly to Luma. No ADB install or accessibility repair was used for that final upgrade. Prior temporary configuration changes from failed installation-method tests were restored.
The source classification follows Android's [PackageInstaller source definitions](https://developer.android.com/reference/android/content/pm/PackageInstaller#PACKAGE_SOURCE_OTHER); the platform's [file-source restriction implementation](https://android.googlesource.com/platform/frameworks/base/+/143db55921dd%5E%21/) explains the behavior encountered during the earlier file-install tests.
## Integrated TV rendering — October 19, 2026
The authorized second device (`192.168.50.200:5555`) is a TCL Smart TV Pro / G08, Android 14 (API 34), ARM Cortex-A73 cores, Mali-G52 OpenGL renderer, approximately 2.34 GiB physical RAM. Its existing 1920 × 1080 render override and density 320 were retained. About 1 GiB was available at diagnosis; Android reported no thermal throttling. The original installed version was 1.1.5.
A repeatable ADB D-pad sequence on 1.1.5 reported 627 rendered frames, 310 missed frame deadlines (49.44%), median 57 ms and 90th percentile 69 ms in `dumpsys gfxinfo`. Frame timing was recorded on the TV itself and is independent of ADB/Tailscale latency.
A separate same-signature `PerformanceInstrumentation` diagnostic isolates the scene and tile shadows. It requests the same first-six-tile sequence 36 times with 350 ms between selections; this includes moving to/from the next row. Android `FrameMetrics.TOTAL_DURATION` includes renderer/queue waits and differs from the histogram summary in `gfxinfo`. These are comparable A/B samples from the same TV, rather than benchmarks across different devices:
| Rendering variant | Frames | Median | 90th percentile | Frames over 16.7 ms |
| --- | ---: | ---: | ---: | ---: |
| 1.1.5 unchanged | 433 | 85.97 ms | 89.50 ms | 433 (100%) |
| 1.1.5 background hidden | 366 | 9.22 ms | 10.46 ms | 0 |
| 1.1.5 tile shadows removed, background restored | 433 | 84.90 ms | 87.39 ms | 433 (100%) |
| Precomposed background, original shadows | 638 | 14.40 ms | 25.79 ms | 185 (29.0%) |
| 1.1.6 background and focused shadow enabled | 639 | 12.99 ms | 17.00 ms | 64 (10.0%) |
| 1.1.6 shadows removed for diagnosis only | 640 | 12.67 ms | 13.59 ms | 0 |
The full-screen shader/transparency passes were the dominant bottleneck on this renderer. Removing redundant shadows after fixing that bottleneck reduced the remaining tail latency. This supports an application rendering optimization issue exposed by this TV's graphics implementation; there is no evidence here that replacing the TV or adding RAM is necessary. This is a short controlled navigation sample, not a claim of universally perfect 60 fps.
Version 1.1.6 composes the original artwork, accent glow, darkening, and gradient protection on one background worker into opaque textures no larger than 960 × 540. Each texture is about 1.98 MiB. The worker has one pending-request slot, discards stale results, and times out after two idle seconds. Navigation crossfades the finished textures rather than regenerating full-screen shaders each frame. Only the focused tile casts a shadow, and its parent no longer casts a duplicate shadow. Focus enlargement, smooth scrolling, app-colored accents, and user preferences remain enabled. This follows Android's guidance on [reducing overdraw](https://developer.android.com/topic/performance/rendering/overdraw).
The enhanced wake regression passed cold launch, interrupted loading followed by pause/resume, memory trim recovery, existing-image reuse, partial-grid recovery, composed-texture resolution/nonblank checks, release of all composed/source textures on stop, screensaver dismissal resuming rendering, and five stop/start cycles. No preferences were rewritten by the diagnostic. Temporary test APKs are not part of the published launcher.
Raw local captures are in `output/performance/` (ignored by Git), including the original screenshot, A/B timings, final navigation frames, memory/idle samples, and final screenshot.
The original D-pad workload was then repeated against the final installed APK: **853 frames, 12 missed deadlines (1.41%)**, median 13 ms, 90th percentile 24 ms, and no slow bitmap uploads. The original missed-deadline rate was 49.44%. Screenshots retain the mountain artwork, focus enlargement, five-column layout, and partial second row. The difference from the isolated benchmark reflects using actual D-pad input and more repeated scrolling.
Foreground PSS was 43,742 KiB (42.7 MiB), and background PSS was 35,803 KiB (35.0 MiB). The launcher consumed zero CPU ticks over five seconds while Android settings was foreground. Source/composite caches are cleared when stopped; actual heap reclamation remains Android's responsibility. These short samples do not establish overnight endurance.
The TV's Home override was already off (no enabled accessibility services) and was retained. A physical Home-key injection returns to Google Home on this TV; opening Luma directly and repeated Back stays in Luma. No default-launcher, picture, power, resolution, or accessibility settings were changed. The optimized APK was installed only on the authorized TV at `.200`; the onn was not operated during this investigation. Version 1.1.6 is published in the existing Git update feed for the other devices.
## Further focus optimization and onn comparison — October 19, 2026
The onn was still running 1.1.5. After waking it and allowing its images to settle, the same 42-key D-pad workload reported 603 frames, 34 missed deadlines (5.64%), median 20 ms, 90th percentile 23 ms, and 95th percentile 24 ms. Thus the integrated TV's 1.1.6 results were already comparable to the working onn installation, with a lower median and missed-deadline rate. This compares the devices' actual installed versions and app grids; it is not a same-version hardware ranking. The onn's package, preferences, and Home override were not changed; its original standby state was restored after measurement.
On the TV, the 1.1.6 diagnostic showed that the remaining native focused-tile shadow increased tail latency. Version 1.1.7 replaces it with one shared 192 × 122 ARGB soft-shadow texture (93,696 bytes), tinted for each app and faded over the existing 160 ms focus transition. The existing 6.5% focus scale, glow, background colors, and scrolling remain enabled. This avoids animating live elevation-shadow geometry and does not require reducing image resolution further or removing animations.
The same TV-local focus benchmark with the new shadow enabled produced **640 frames, median 13.48 ms, 90th percentile 14.31 ms, and zero frames over 16.7 ms**. Version 1.1.6's corresponding sample was 639 frames, median 12.99 ms, 90th percentile 17.00 ms, and 64 frames over 16.7 ms. The gain is consistency/tail latency rather than a faster median. Disabling the cached shadow for diagnosis barely changed the result (median 13.33 ms, 90th percentile 14.10 ms), indicating its remaining rendering cost is small.
The enhanced wake regression passed with the cached-shadow build, including checking that its animation stops on leaving Home, nonblank bounded backgrounds, interrupted-load recovery, screensaver dismissal, and five stop/start cycles. An initial attempt began while the TV was asleep; later ADB disconnects interrupted streamed test reports. Running the diagnostic locally with saved output while the TV was awake completed successfully. The temporary diagnostic APK is removed after verification.
The broader 42-key D-pad sequence on 1.1.7 recorded 861 frames, 16 missed deadlines (1.86%), median 14 ms, 90th percentile 25 ms, and 95th percentile 25 ms. This is roughly level with the 1.1.6 sample (1.41%, 13/24/26 ms); the cached shadow's isolated-focus consistency gain did not produce a measurable improvement to the whole navigation workload in these samples. It is not presented as a new overall performance leap. Screenshots confirm the soft app-colored shadow, scale/outline, wallpaper, and spacing remain visible. The remaining navigation time is primarily GPU/compositing work; main-thread draw/traversal was already sub-millisecond in the sampled 1.1.6 trace. The onn comparison remains 1.1.5, rather than claiming a same-version hardware comparison.
## Portable tile optimization — October 19, 2026
Version 1.1.8 uses the same default rendering path and settings on every device. No model, manufacturer, product, IP address, or device-name selection is used. App banners are marked opaque after composition over an opaque background. Rounded app artwork uses a cached bitmap shader and matrix, avoiding repeated path clipping and redundant background draws. Transparent artwork retains its underfill and media progress bars retain their rounded clipping. The shared focus shadow is now ALPHA_8: 23,424 bytes instead of 93,696 bytes. Focus enlargement, color glow, 160 ms focus transitions, background transitions, and smooth scrolling remain enabled.
The onn and integrated TV were first brought to the same released 1.1.7 baseline. The following comparisons use a same-signature diagnostic with real D-pad key injection, Android FrameMetrics, and identical 42-key sequences across the app rows. Normal navigation waits 350 ms between keys; rapid navigation waits 80 ms. This avoids the extra process/VM startup caused by invoking Android's shell `input` command for every key. Metrics were collected locally on each device, rather than timing the ADB connection.
| Device / pace | 1.1.7 median / p90 | 1.1.8 median / p90 | Missed deadlines, baseline → optimized |
| --- | ---: | ---: | ---: |
| onn, normal | 12.09 / 15.08 ms | 10.49 / 13.30 ms | 8/798 → 7/801 |
| onn, rapid | 11.60 / 14.89 ms | 10.76 / 13.63 ms | 3/254 → 1/255 |
| TV, normal | 25.03 / 26.75 ms | 13.28 / 24.24 ms | 32/828 → 17/864 |
| TV, rapid | 25.36 / 26.91 ms | 24.39 / 25.17 ms | 28/212 → 25/210 |
The TV improves substantially in normal-navigation median duration, but its rapid-navigation GPU cost remains greater than the onn's. These are short samples of each device's existing app grid, not proof of identical hardware performance or perfect 60 fps. FrameMetrics total duration and deadlines are reported directly; they should not be mixed with earlier gfxinfo histogram values or an isolated first-six-tile benchmark. All samples above had zero dropped FrameMetrics callbacks. Raw reports are saved locally under `output/performance/round2/` and are ignored by Git.
**Settings → Appearance → Memory saver** is available on every device and is **off by default**. It separates accent tint into a bounded alpha mask, retaining the same composed background when only the selected app's color changes. This reduces repeated full-image allocation/uploads, with a different GPU tradeoff. Earlier shell-workload onn PSS samples were 132,417 KiB for 1.1.7, 109,413 KiB for optimized tiles with the default background, and 67,578 KiB with Memory saver and optimized tiles. PSS is variable, includes framework/graphics overhead, and is not a guaranteed reduction on every device.
With Memory saver enabled, the same native-input test measured normal/rapid p90 durations of 13.03/14.88 ms on the onn and 24.44/24.81 ms on the TV. Missed-deadline counts were 12/681 and 1/259 on the onn, and 20/732 and 27/208 on the TV. It is not consistently faster than the default, so it is an explicit preference rather than an automatic device classification. Both modes work on both devices. GPU-specific single-pass shader experiments were discarded after they regressed performance.
GPU PixelCopy checks passed on both devices for wide opaque banners, portrait center cropping, transparent artwork, media progress bars, and rounded corners. Sampled tile interior differences against the previous drawing implementation were at most 1/255 per color channel. Clearing a tile image also clears its cached shader reference. Memory-saver backgrounds matched the original composited colors within 2/255, and artwork crossfades at 25%, 50%, and 75% matched within 3/255. Accent changes reused the existing background and mask.
Wake regression passed on both devices in both rendering modes: cold launch, interrupted loading/pause/resume, memory trim, existing-tile reuse, partial-grid recovery, bounded nonblank backgrounds, texture/queue release on stop, screensaver dismissal, and five stop/start cycles. Memory-saver checks also verified loading the saved preference and switching back and forth at runtime. Temporary preference changes were restored by the test runners. The first tile-comparison test had an uninitialized reference bitmap in the diagnostic overlay; fixing that test setup allowed the comparisons to pass without changing production rendering.
The final signed 1.1.8 APK was installed on both devices and retains the previous certificate. Final screenshots show the original five-column spacious layout, mountain background, rounded banners, and app-colored focused shadow. Temporary diagnostic packages were uninstalled. The onn Home helper was re-bound after instrumentation using its original enabled-service list, retaining Button Mapper; Home from Android settings and repeated Back returned/stayed in Luma. The TV's Home override remains off with its original empty service list. Projectivy remains installed and enabled.
With Android settings foreground, the onn UI and Home helper each consumed zero CPU ticks over five seconds. The TV initially consumed five CPU ticks over five seconds after installation; a later five-second sample after settling consumed zero. These short samples show idle behavior after pending work completes, rather than establishing overnight endurance. Both devices were returned to Luma after verification.