AstroFrame/docs/analysis-design.md
Elijah c0ef081e4d
All checks were successful
Release AstroFrame / packages (push) Successful in 56s
Release 1.1.7: reject moderate thin cloud and confirmed star elongation
Frames the sky-structure detector already flags become REJECT when
strong cloud texture is backed by two independent moderate dimming
measurements. Elongation confirmed on the same matched reference stars
(lower 3-sigma bound of the change >= 0.10) rejects as a roundness
outlier, following Siril's Seestar -filter-round practice. NGC 1432:
Review 36 -> 1; every Keep unchanged across the validation corpus.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:48:07 -07:00

74 lines
16 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Analysis design and architecture
## Evidence and scope
The supplied 1,920 originals cover four targets and five sessions, with 2160×3840 GRBG Bayer data, 30-second exposures and two filter settings. Original FITS files are opened read-only. This application evaluates and exports lights; Siril handles calibration, registration, stacking and post-processing.
The first algorithm failed real owner-reviewed examples: cloud structure inflated detected-star counts; flux only sampled surviving detections; rolling baselines absorbed prolonged cloud; a small ellipticity deviation reviewed a clear image. The v2 design replaces the cloud measurements, reference construction and classification policy. Rejection counts are not a target or a validation criterion.
Version 1.1.1 refines this policy with geometric fit diagnostics, local cloud-aperture corroboration, and matched-star morphology checks. Version 1.1.6 adds thin-cloud texture with same-exposure differential photometry; 1.1.7 adds moderate-cloud corroboration and confirmed-elongation rejection. The original v2 results remain a baseline; see [the 1.1.1 comparison](algorithm-1.1.1-validation.md) for the measurements and limits of the new validation.
## Components
- **Engine:** FITS reader, independent `IFrameDetector` measurements, background reference construction, aperture photometry, named decision rules, persistence and export. No GUI, Python runtime, AI or GPU dependency.
- **Desktop:** Avalonia/.NET 8 on Windows and Linux; asynchronous orchestration, graph inspection, spatial overlay and manual labels. Windows software rendering and native window decorations.
- **Validation:** CLI analysis, real labeled regressions, provisional fixed-time visual samples and synthetic image invariants. Stack quality remains a separate validation task.
Additional detectors can add measurements independently. A future AI reviewer can consume previews/measurements and supply evidence without replacing deterministic analysis or human overrides. GPU acceleration remains optional and unimplemented.
## Measurements and reference
1. Read one FITS image at a time. Bayer data use equal-phase 2×2 averages. A 24×40 grid measures a clipped median sky and difference-based noise. The local interpolated sky, rather than a coarse global region, sets detection significance. A local annulus verifies that a peak rises above its own background, rejecting diffuse cloud texture. Core moments estimate FWHM and ellipticity; they are not calibrated Siril PSF fits.
2. Group by target, camera, exposure, filter, gain, dimensions and Bayer pattern; split by timestamps, gaps over two hours and coordinate changes. Each session has its own reference.
3. Select an anchor from low-curvature backgrounds with representative star population and morphology. Bright-star pairs estimate a similarity transform, then hundreds of correspondences refine rotation/translation. Maps and reference aperture coordinates follow that transform. Field edges outside the overlap remain unmeasured.
4. Robustly remove each map's additive offset and planar gradient. Form an aligned median background map, measure transient residuals, and rebuild three times from the least transient 40% of exposures. This fraction constructs a reference; it is **not** a rejection quota. Persistent target nebulosity belongs to the reference. Smooth sky gradients and offsets are removed rather than interpreted as cloud.
5. Read each exposure once more, sequentially, for signed aperture photometry at up to 600 reference coordinates. A local annulus subtracts sky. Reference brightness establishes aperture eligibility; a star need not be detected in the tested exposure. Median fluxes from reference exposures normalize global and 3×4 regional transparency. This addresses survivor bias. Saturated apertures and positions outside the field are excluded explicitly.
6. Compare 960 aligned sky cells with the reference. Cell scatter comes from reference-frame residual MAD with a floor relative to measured image noise. Store a mask above five times that scatter, affected overlap fraction, largest connected component, 95th-percentile significance, residual ADU, aperture counts and reference overlap. These are named evidence, not an opaque combined score.
7. Measure thin-cloud texture separately. Remove a robust quadratic surface, not only a plane, from each aligned residual, then average each cell with its 3×3 neighbors. Divide by the per-cell scatter that reference exposures show for the same smoothed statistic; the floor is half the median of those scatters. Cells above four times scatter form a texture mask. Store the 98th-percentile texture significance, affected and largest connected fraction. Compare the median forced flux of reference apertures inside the mask with apertures in clear cells (below twice scatter) of the same exposure. Normalize that inside/outside ratio by the same apertures in the nearest 40 reference exposures and divide its loss by their MAD, with a 1% floor. Global transparency, seeing and airmass changes cancel in this differential ratio; a sky-only gradient leaves it near one.
Grid background estimation and local aperture subtraction are standard image-analysis techniques; see [Photutils background documentation](https://photutils.readthedocs.io/en/stable/api/photutils.background.Background2D.html). This implementation is independent C#, using the supplied data to establish its comparison distributions.
## Classification
Morphology and flux baselines use the nearest 40 low-transient reference exposures, excluding the tested frame. Cloud-contaminated neighborhoods do not train their own normal baseline. This follows gradual changes where clear comparison frames exist while retaining a wider session reference during cloudy gaps.
REJECT evidence routes are explicit:
- Strong spatial structure: at least 25× reference scatter at the 95th percentile, at least 8% affected overlap and a connected area of at least 3.5%. A lone sky signal must be overwhelming before discarding integration; weaker coherent structure needs stellar corroboration or routes to REVIEW.
- Moderate coherent spatial structure plus substantial global flux loss, regional forced-flux loss or detected-star loss.
- Coherent structure at least 8× reference scatter, at least 8% affected overlap, and a connected anomalous area of at least 3.5%, corroborated by positive excess-sky coverage of at least 8% and a positive connected area of at least 3.5%. At least 20 reference apertures in two regions inside that excess sky must show below 90% of nearby same-aperture flux, with at least three times the larger of reference scatter, a 3% floor, and propagated local photometry uncertainty. Affected-aperture comparisons use the nearest 40 reference members, excluding the tested exposure, and require at least 12 valid comparison frames.
- Thin cloud on an exposure already routed to REVIEW by another measurement: texture significance at least 10, a connected textured area of at least 5%, at least 20 apertures both inside and outside the texture, and stars inside at most 98% of their clear-cell flux with a loss of at least 2.5 times nearby reference scatter. Applying real cloud masks to clear exposures put 0.4–0.6% of differential deviations at or above 2.5, and clear exposures do not reach the texture prerequisite. This route never rejects an otherwise clean KEEP.
- Moderate thin cloud on an exposure the planar structure detector already flags (at least 5× scatter over 2% of the overlap): texture significance at least 20 with the same connectivity and aperture requirements, inside/outside dimming of at least 1.5% and 1.5 times scatter, and independently at least 20 excess-sky apertures in two regions at or below 93% of nearby same-aperture flux with at least 2.5 times scatter.
- Severe global flux loss (below 60% of clear-frame baseline) with substantial detected-star loss.
- Severe combined stellar width and ellipticity degradation, or almost complete star disappearance relative to a rich reference field.
- Confirmed elongation: an existing elongation flag (at least four scatters and 0.15 above the nearby ellipticity baseline) that the same matched reference stars confirm. This requires at least 50 matched stars and 12 nearby reference frames, uncertainty at most 0.025, and a median change whose lower three-standard-error bound is at least 0.10, the same limit below which a flag is dismissed. Changes between the two limits stay REVIEW. Stacking workflows discard such roundness outliers; for example, Siril's Seestar script applies `-filter-round=2.5k`. On both supplied NGC 1432 nights, 22 of 38 confirmed frames fall below Siril's own roundness cutoff and the rest lie near it. Every confirmed frame exceeds an eccentricity-equivalent of 0.6. Unconfirmed or imprecise flags remain REVIEW.
Automatic rejection requires enough comparable frames, valid time and adequate aligned overlap. Ordinary spatial rejection also requires verified geometry: at least 20 tight correspondences, at least 80% of correspondences within the width-scaled tight tolerance, three regions with at least three tight matches each, and a spread of at least 15% of the image width and height. The median residual must be at most the larger of 0.5 original pixels and 0.2× reference width; the 95th percentile must be at most the larger of one original pixel and 0.5× reference width. Transform scale must remain between 0.975 and 1.025. Missing stars alone no longer veto a cloud rejection when geometry and aperture coverage are verified; low star survival still cannot certify KEEP. Catalog-only classification retains its previous match-fraction safeguard.
An existing elongation flag can be dismissed only when at least 50 matched bright reference stars and 12 nearby reference frames provide a precise, small shape change. Its median change plus three estimated standard errors must be below 0.10, and the uncertainty must not exceed 0.025. Missing pairs cannot dismiss a warning. Matched stars use the same reference identities rather than independently selected detection populations. This remains moment-based photometry, not a new PSF-fitting implementation. Ambiguous structure, moderate flux loss, substantial single-metric morphology changes and uncertain measurements route to REVIEW. Minor ellipticity fluctuations and a high global background alone do not trigger review.
These significance and effect-size limits are **engineering policy, not calibrated probabilities or guarantees**. Reference scatter, noise, star population, flux, seeing and background are measured separately per session. No fixed acceptable star count or absolute sky brightness is used. The score exists only for sorting.
## Limits that need validation
A wholly uniform, cloudy session may lack any internally observable clear reference. Clouds that resemble a smooth gradient and have little detectable attenuation are difficult. Dense nebulosity, alignment failures, crowded fields, large drift/rotation and changing PSFs can bias maps or fixed-aperture flux. The main structure model removes only a planar sky trend; a legitimate time-varying nonlinear sky pattern can require review. The thin-cloud texture also removes quadratic curvature, but requires stellar dimming beneath it before rejecting. Faint cirrus on an otherwise clean exposure remains KEEP: its effect on a stack has not been validated. The red overlay identifies anomalous sky structure, not a pixel-accurate physical cloud segmentation.
Small sessions and unreliable references cannot automatically reject. Reference stability also checks reference-map residuals against measured pixel noise; a wholly unstable comparison population routes to REVIEW. This does not detect a perfectly uniform, consistently attenuated session. Human overrides remain separate from automatic results. Do not claim the method matches manual Siril selection until independently processed stacks support that claim.
## Memory, caching and safety
Version 1.1.0 selects up to 16 workers from CPU capacity, leaving two logical processors unused where possible. A 48-frame measurement run on a 16-logical-processor machine took 2.93 seconds with four workers and 1.32 seconds with fourteen, with identical star counts (peak working sets 251 and 762 MiB). This measures stellar/background processing, not whole-project speedup. No full-resolution image collection or stack is held in RAM. Windows uses a verified process ceiling based on physical RAM minus its reserve, rounded to an allocation boundary. Fixed 6 GiB process, 3 GiB heap and 256 MiB frame caps have been removed. Per-frame estimates include a 64 MiB working margin and are limited to half the machine budget. Both image passes, previews and star detail acquire atomic memory reservations, sampling physical headroom for every frame. The OS/desktop reserve remains one quarter of RAM up to 8 GiB; page-file capacity does not satisfy it. Sustained external pressure stops safely after active frames finish and saves completed measurements.
Small JSON metadata stores decisions, notes, evidence and compact masks. Binary versioned caches hold stars, sky maps and forced photometry, validated against source size and modification time and reference identity. A new engine version invalidates analysis without losing human labels. Repeated analysis uses cached measurements and forced apertures when valid. Project writes replace a temporary file atomically. Cancellation leaves analysis incomplete and export disabled.
Display caches have a separate version. They contain compressed, linear, area-averaged thumbnails rather than a permanently stretched byte image. **Frame view** retains its noise-aware black point and adjustable median brightness. **Denoised** is the only alternative. It performs non-local means on the linear thumbnail using 5×5 patches, a 13×13 search window, reflected borders, noise-bias subtraction and a filter strength of 0.8 times the estimated noise sigma. Its black point, white point and normalized median come from the original thumbnail; they are never recomputed from the denoised result. This prevents the stretch from amplifying residual noise after denoising. It adds 0.07 to the requested display median, matching the approved samples' default brightness of 0.25 at slider value 0.18.
The previous Nebula, Asinh and Color nebula modes are removed. The denoised result stays in the selected viewer's memory and is computed only when requested. It needs no new disk cache or FITS reader and works on monochrome and Bayer-derived grayscale thumbnails alike. Existing linear caches remain compatible, and existing color caches are simply unused. The analysis path, classifications and original FITS files are unaffected.
Prepared display statistics and the denoised plane are retained in the viewer so slider changes remain responsive. Loading and denoising run off the UI thread, with cancellation, atomic memory reservations and a single preview-loading gate preventing queued work during rapid navigation. The original Frame preview remains available while denoising runs. Both comparison panes use the selected mode and shared zoom/pan. Old projects regenerate only requested displays on demand. Denoising is a viewing aid and may soften weak detail; Frame view and source-star inspection remain available for judging unfiltered morphology.
The preview is not an input to classification. Source-star inspection reads two originals, maps identical stars through their stored alignment, and displays enlarged source superpixels without interpolation. Crops are independently normalized to judge morphology. See [Siril's screen-transfer discussion](https://siril.readthedocs.io/en/latest/processing/stretching.html) for the distinction between display stretching and linear image data.
The desktop allows one instance. UI tests run sequentially. The earlier native memory-access crash has not received a confirmed root-cause diagnosis; memory enforcement and later successful runs must not be confused with that diagnosis.
Export validates source fingerprints, refuses source/output-tree overlap and nonempty destinations, never overwrites original FITS, and supports hardlinks or bounded byte-for-byte copy fallback. Hardlinks share source bytes; downstream tools must write separate outputs. Export separates different targets and retains an audit manifest with classifications and cloud/flux measurements.