Inital Version
This commit is contained in:
commit
94cd6b984b
211 changed files with 4765 additions and 0 deletions
27
docs/validation-protocol.md
Normal file
27
docs/validation-protocol.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
# Validation protocol
|
||||
|
||||
No claim of matching or exceeding manual Siril selection is justified yet. Five owner-reviewed examples now supply real regression labels; they are not a representative accuracy sample. An additional 56 fixed capture-order previews provide a provisional cloud-only screen. No controlled stacks have been produced.
|
||||
|
||||
## Frame-level verification
|
||||
|
||||
1. Review automatic REJECT frames and the REVIEW pool. Keep automatic decisions separate from human overrides. The v1 count of five REJECT frames was a failed result, not a reference target.
|
||||
2. Sample KEEP exposures throughout every session, including session edges, low star-count stretches, elevated background periods and transitions. A review confined to REJECT cannot measure missed bad frames.
|
||||
3. Record human KEEP/REVIEW/REJECT and notes. For meaningful statistics, finish labeling a session or use a clearly documented representative sample.
|
||||
4. Run the label summary. Report false rejection of human KEEP in frames and exposure seconds, automatic KEEP of human REJECT, ambiguous cases, confusion counts and labeled/unlabeled coverage per session. Human REVIEW is uncertainty, not a definite good/bad label.
|
||||
5. Never infer that an unlabeled frame is a correct decision. Retain a held-out target/night before tuning policy rules.
|
||||
|
||||
The cloud-specific regression loop is `python tools/check_real_frames.py projects/Sample\ Survey/project.json` (quote the path for your shell). `tests/real-frame-labels.json` records the owner's exact examples. `tools/check_visual_holdout.py` evaluates `tests/visual-cloud-holdout.json`; clear-looking previews only assert no automatic rejection, since preview screening does not validate PSF or uniform attenuation. Synthetic image regressions include persistent nebulosity, gradual gradients/flux drift, and a prolonged moving cloud stretch. None is a substitute for independent owner labels and stacks.
|
||||
|
||||
## Controlled Siril comparison
|
||||
|
||||
Prepare three input cohorts per target/session:
|
||||
|
||||
- All originals.
|
||||
- Human-selected lights, with a documented policy for human REVIEW.
|
||||
- Automatically selected lights, with a documented policy for automatic REVIEW. Keep-only and keep-plus-review are distinct experiments.
|
||||
|
||||
Use the same calibration files, debayering, registration/reference, output geometry, weighting, pixel rejection, normalization and subsequent processing. Work from exported copies or read-only inputs into separate output folders. Do not optimize one cohort independently. Log selected frame names and total integration before processing.
|
||||
|
||||
Compare registered crops using the same display stretch and measurement regions: background noise, faint-feature signal-to-noise, stellar size/ellipticity, gradients/cloud residue, color stability, artifact rates and retained integration. Include both equal-processing results and an equal-integration sensitivity check where appropriate. Judge the nebula/galaxy and stars, not just a global sharpness statistic. Preserve output stacks and settings so the comparison is reproducible.
|
||||
|
||||
The automatic method can be claimed comparable only after these outputs support that conclusion across targets and nights, with an acceptable false-rejection budget agreed from the results. No fixed acceptance budget is invented here.
|
||||
Loading…
Add table
Add a link
Reference in a new issue