camera-based-rgbww-optimizer

name	camera-based-rgbww-optimizer
description	Use when working on camera-based RGBWW calibration, RGBWW channel control, Canon camera capture, Ray120c reference matching, or color-chart workflows.

Camera-Based RGBWW Optimizer Skill

Project Facts

• Planned optimizer language: Python.
• Implementation package: src/camera_based_rgbww_optimizer/.
• Source layout: control/ for device control, optimization/ for measurement/calibration workflows, utils/ for shared helpers and post-processing, and interaction/ for local UI tools.
• Root-level compatibility scripts are intentionally not part of the current interface.
• ESPHome device name and default host: bedroom-rgbww-strip / bedroom-rgbww-strip.local.
• ESPHome firmware configs live under firmware/esphome/bedroom-rgbww-strip*.yaml.
• .venv/bin/python -m pip install -e . installs the package's Python dependencies; gphoto2, exiftool, and dcraw_emu remain system dependencies.
• Reference light: Aputure Amaran Ray120c at 100% brightness.
• Camera path: Canon EOS R6 Mark III over USB PTP using gphoto2.
• All new formal experiments use control.camera_remote.RemoteCameraBackend; local gphoto2 remains compatibility/test-only. camera_remote health --json performs only unauthenticated GET /health, while config --json performs only authenticated GET /v1/config; neither captures or changes settings. Only the Python process reads URL/token from environment or project .env; never source, print, pass, or persist the token. Capture retries camera_busy with a bounded whole-request retry, atomically saves CR3, and records backend/endpoint/attempt count without credentials. camera_gphoto2.auto_expose_capture owns the shared exposure search and receives the same injected backend for trials and final capture.
• On macOS, camera_gphoto2 retries once after a gphoto2 USB-claim or kernel-driver-detach failure by killing per-user ptpcamerad/icdd Image Capture processes; set CAMERA_BASED_RGBWW_OPTIMIZER_DISABLE_MACOS_PTP_RECOVERY=1 to disable that recovery during host diagnostics.
• Measurement basis: fixed linear camera RAW RGB pipeline by default. Visual fit-check tools default to the camera-rendered JpgFromRaw preview and may explicitly use dcraw_emu preview decoding as a fallback.
• Calibration scene: black card, white card, 18% gray card, 24-color chart, RGBWW output, and Ray120c output.

Workflow

1. Check docs/README.md before changing calibration or device-control behavior.
2. Use package module commands, for example .venv/bin/python -m camera_based_rgbww_optimizer.control.camera_gphoto2 ....
3. Keep all temporary project outputs under repository tmp/.
4. For channel tests, use low initial values and test one physical channel at a time.
5. For Canon R6 Mark III sessions, use camera_based_rgbww_optimizer.control.camera_gphoto2.
6. For Ray120c CCT/G-M, HSI, and RGB reference control, use camera_based_rgbww_optimizer.control.ray120c.
7. For explicit cleanup after manual or interrupted runs, use camera_based_rgbww_optimizer.control.all_off; it calls ESPHome all_off and sets Ray120c intensity to 0.
8. For RGBWW code/duty response measurement, use camera_based_rgbww_optimizer.optimization.measure_channel_response.
9. For color-block location annotation, use camera_based_rgbww_optimizer.interaction.location_picker_ui; add --camera-backend remote to run its complete auto-exposure and capture sequence through r6-camera-bridge. It keeps linear .npy RAW only for auto-exposure metering, then uses the camera-rendered JpgFromRaw preview for the Web UI image and automatic block detection. Out-of-bounds automatic detections are rejected instead of shown.
10. For checking a saved location JSON against a fresh camera capture, use camera_based_rgbww_optimizer.interaction.location_checker <locations.json>; it uses full-frame auto-exposure, extracts the camera-rendered JpgFromRaw preview from the final RAW, and only uses the location JSON for outline + color-name label fit checking.
11. For offline native Bayer patch measurement, use camera_based_rgbww_optimizer.measurement.raw_mosaic_patches RAW --exposure SHUTTER. It reads raw_image_visible directly, maps the saved color-chart coordinates to the visible mosaic, keeps R/G1/G2/B separate, optionally derives G, and can normalize/subtract a separately exposed ambient RAW. It does not call the camera, Ray, or ESPHome and does not create .npy files. Acquisition runners must resolve physical exposure through measurement.recorded_exposure: prefer CR3 Canon:ExposureTime, fall back to standard EXIF, and treat requested shutter labels only as a provenance-marked degraded fallback. Quality gates come only from config/camera/raw-observer.json. P1 validated identity RAW only in the 1.0x..1.25x per-source auto-exposure envelope, with zero clipping/nonfinite tolerance, near-noise fraction at most 0.05, at least 29000 valid pixels per plane, and combine_green=false; a rejected weak feature must be reacquired at a weak-signal exposure rather than retained by widening thresholds.
12. For WW/CW endpoint CCT/G-M detection, use camera_based_rgbww_optimizer.optimization.measure_white_endpoints; this is a separate reusable calibration step and should usually run before formal white LUT optimization. Its default mode photographs strip WW/CW endpoints, estimates white-patch CCT/Duv from linear RAW, then controls Ray120c through target-driven alternating CCT/G-M correction: start from the strip-estimated CCT, correct the next CCT setting from observed error, solve G/M from observed Duv error, and repeat for three rounds. Defaults are total budgets of up to 64 CCT samples and 16 G/M samples per endpoint. Use --ray-start ww=CCT,GM --ray-start cw=CCT,GM to seed from a previous run. Use --ray-search-mode grid only when the fixed Ray reference grid is needed for cross-checking.
13. For formal CCT/G-M white LUT calibration, use camera_based_rgbww_optimizer.optimization.measure_white_lut --camera-backend remote --location-config config/location/locations-20260714-211759.json. The canonical endpoint input is config/white/white-endpoints.json, the default Ray grid is 1800, 3200, 5600, 20000K x raw G/M 0, 100, 200, and the only new-experiment location has SHA-256 21f02328bff8b5ffcfc10c767e15cdefbf8bb883bc352eff6f0fb7ade38a8f80. Every capture must have exactly one source on: turn the strip off before enabling Ray, then set Ray intensity to 0 immediately after its reference capture and before enabling the strip. Maintain independent Ray and strip automatic-exposure shutter caches; each target's initial strip candidate auto-exposes from the previous strip result, while its probes and feedback keep that resolved strip shutter fixed. Treat every linear .npy decode as temporary: delete it after region statistics, white-point estimation, and exposure checks for Ray, initial strip, probe, feedback, and confirmation captures; retain RAW files, decode metadata JSON, and calibration results. The channel-basis solver supplies an initial high-output command; --initial-lut may replace it with a previous measured command for strict repeat passes. Outside the native CCT span, low-penalty compensation RGB channels may own the required 4095 high-output constraint so the endpoint white channel can move away from saturation. Camera-observed CCT/Duv feedback then probes the four free channels and estimates a local Jacobian. The measured default goal is Duv below 0.0005 plus a dynamic CCT tolerance of at least 25K, raised at high CCT to the Kelvin equivalent of 1 mired. When measured samples bracket the target CCT and interpolated Duv remains within tolerance, use sample-pair secant interpolation for the next command; otherwise run up to sixteen constrained Broyden correction rounds. Solve each damped Jacobian step with channel headroom, the 0..4095 range, and the round step limit included as box constraints; never solve unconstrained and clip afterward because clipping can discard compensating terms at saturated channels. A candidate only completes after two independent off/on reapplications of the same command also meet the goal; each confirmation holds the strip off for one second, then holds the command for 60 seconds before capture. A failed settled confirmation becomes a new thermal base, the local Jacobian is rebuilt, and older transient samples no longer drive secant prediction. Two consecutive materially inaccurate bracketed-CCT predictions trigger the same rebase so an early cold sample cannot trap later warm-state secant steps. RGB penalties are computed per R/G/B channel from both CCT extension and distance from the endpoint-interpolated native G/M: red is released for warmer extension, blue for cooler extension, red plus blue toward magenta, and green toward green. For strict reruns, compare the previous measured command's score against the white-patch camera-RGB cone seed: keep the previous command first when its strict score is at most 4, otherwise start from the cone model, and retain the other as fallback. Exhausted attempts may restart with the endpoint-biased or model-selected full-scale channel active; attempt indices must remain part of capture filenames.
14. To apply a completed white LUT, use camera_based_rgbww_optimizer.control.white_lut CCT GM. It reads config/white/white-lut.json by default, performs bilinear interpolation over neighboring targets using Kelvin on the CCT axis and linear raw G/M on the tint axis, applies optional linear --intensity-percent scaling, and delegates the final command to the shared ESPHome controller. It is an explicit light-on tool and intentionally leaves the strip on.
15. For RAW-observer validation, first use camera_based_rgbww_optimizer.optimization.measure_raw_observer_validation --baseline-lut PATH --camera-backend remote --location-config config/location/locations-20260714-211759.json --target-max 19661 --linearity-factors 0.5,1,1.5 --preflight-only --dry-run. It requires an external best 70-point LUT and writes an immutable five-capture plan without loading .env, network access, or light calls. The plan freezes backend, location path/hash, and Python/shell cleanup. Real P1 additionally requires --execute --allow-high-output, explicit user hardware approval, and tmp/run-raw-observer-validation.zsh; trials and final captures reuse one backend, recorded exposure remains mandatory, and all failures enter Python plus shell cleanup.
16. After P1 completes, run camera_based_rgbww_optimizer.optimization.analyze_raw_observer_validation RUN_DIR. It restores source/purpose/repeat/factor/shutter provenance from the run records, then emits source-specific repeatability and linearity reports plus G1/G2 context. For legacy runs, add --recompute-recorded-exposure; this verifies frozen config hashes, reprocesses retained CR3 files, preserves analysis.json, and writes analysis-recorded-exposure-v1.json. Review total response and global-scale-removed patch/CFA shape separately. It rejects incomplete runs by default and never declares P1 pass/fail or writes observer thresholds automatically; apply the explicit branch criteria in .agent-work/PLAN.md.
17. If corrected P1 evidence still enters PLAN.md 7.5-D, use camera_based_rgbww_optimizer.optimization.measure_raw_shutter_response for the bounded same-shutter diagnostic. Its plan keeps each source on continuously after one warm-up, uses the same explicit shutter order for Ray and strip, retains native Bayer reports without .npy, and has Python cleanup. Run real acquisition only through tmp/run-raw-shutter-response.zsh so its independent trap writes all-off-shell.json; the runner's offline analysis reports shutter ratios, global per-shutter scale, residual patch/CFA shape, clipping exclusions, exposure provenance, monotonic capture-duration distributions, and recorded source-on commands, but never changes observer thresholds automatically. The conservative Ray default is 1%; ray_intensity_percent is always passed as intensity_percent, while raw intensity is reserved for explicit API-scale values such as shutdown 0.
18. If P1-D reproduces the same shutter-dependent shape residual for Ray and strip, run camera_based_rgbww_optimizer.optimization.analyze_raw_response_candidate FULL_P1_RUN. It fits only the candidate model g(y)=y*(1+a*y+b*y^2), enforces g(0)=0 and a positive derivative on 0..1, records the measured training range with a reject-outside-range policy, and compares it with both identity and a lower exposure envelope. It must not alter native RAW reports or config/camera/raw-observer.json; a complete P1 from another natural day with the same frozen remote inputs is required as holdout before proposing installation. Evaluate that run with --candidate TRAINING_RUN/raw-response-candidate.json --candidate-sha256 FROZEN_SHA256; this mode verifies the artifact hash, rejects the training input hash, refuses out-of-range application, and never refits coefficients. A 7.5-B boundary-only run may add --repeatability-run COMPLETED_FULL_P1 to reuse only that run's paired-repeat noise; location hashes must match and boundary linearity data still come exclusively from the new run.
19. For P2 five-channel basis acquisition, use camera_based_rgbww_optimizer.optimization.measure_chart_response_basis --dry-run first and real runs only through tmp/run-chart-response-basis.zsh. The plan is fixed at 58 captures: ambient 1, per-channel 100% protection and 25% weak auto-exposure baselines 10, anchor baseline 1, cw/ww/r/g/b x 1024/2048/4095 x 3 formal states 45, and one anchor check. It requires remote/latest-location, the code-duty curve, identity RAW, combine_green=false, and the validated 1.0x..1.25x envelope. Analyze only completed runs with optimization.analyze_chart_response_basis; rejected or near-noise observations never enter the basis, and the analyzer reports level-specific matrices/SVD/condition/angles/bootstrap/weights without choosing the Stage-2 branch automatically.
20. The completed P2 basis selected B-B: all five channels use piecewise-linear normalized columns at target codes 1024/2048/4095. Missing low-signal cells may borrow the nearest accepted same-channel level at weight 0.25 only for matrix diagnostics; cells absent at every level are censored zero at weight 0, and neither is treated as direct cross-level evidence. Validate this model with camera_based_rgbww_optimizer.optimization.measure_chart_basis_mixtures --dry-run, then run only through tmp/run-chart-basis-mixtures.zsh. P3 freezes 24 unique commands and two repeats each, with 55 total captures including ambient, protection/weak exposure baselines, and anchors. Analyze only against the frozen P2 artifact and explicit SHA-256 using optimization.analyze_chart_basis_validation; it reports absolute and scale-removed residuals, combined repeat/basis sigma, category/load/order/active-set context, and never selects M-A..M-E automatically.
21. P3 selected M-E / forward_model_seed_only: mixture shape error was broad, not explained by load or anchor drift, and 0/24 samples met the 80%-within-2-sigma target. Do not add a global load correction or interaction expansion. P4 must use the P2 basis only for an initial command, then extend the existing bounded local camera-feedback path with neutral plus multipatch residuals while preserving white-point hard gates, secant/Broyden behavior, and thermal rebase. The frozen feature set is block 02..24 x R/G1/G2/B; neutral is 02..06, training color is 07/09/10/12/13/15/16/18/19/21/23, and holdout is 08/11/14/17/20/22/24.
22. For P4 offline preflight, use camera_based_rgbww_optimizer.optimization.measure_cct_multipatch_pilot --dry-run. It verifies the exact frozen P2/location hashes and writes a 12-target, 92-feature, 84-formal-capture plan without creating a remote backend. The objective uses a nonnegative global brightness scale and strict lexicographic neutral/white/training/holdout ordering. The shared small box-constrained least-squares solver remains the single implementation used by white feedback and multipatch local steps. Real P4 acquisition requires --execute --allow-high-output --camera-backend remote and tmp/run-cct-multipatch-pilot.zsh; the runner rechecks all input hashes, records remote provenance/failure evidence, keeps independent Ray/strip exposure caches, uses native RAW without .npy, and guarantees Python plus shell cleanup. A candidate must pass neutral/white hard gates and rank strictly better than the same-run baseline center; non-converged points still receive two settled final-verification captures.
23. Analyze a completed P4 run with camera_based_rgbww_optimizer.optimization.analyze_cct_multipatch_pilot RUN_DIR. It requires 12 target rows, successful Python and shell cleanup, nonempty CR3 plus recorded-exposure measurement for every capture, and zero .npy. It compares the settled candidate pair with the three-capture baseline center, evaluates training/holdout mean/P90/worst improvement against same-run three-sigma repeatability, checks neutral/white gates and white non-degradation, and reports command continuity/RGB fraction. It never chooses Gate-3 or expands to 70 points automatically.
24. P4 weak-signal handling follows the P1 envelope contract. If and only if every rejected optimization feature is included, has enough valid pixels, passes nonfinite/clipping limits, and fails solely because near-noise fraction exceeds the frozen threshold, reacquire the same source/command once at 1.25x exposure. Keep exposure-normalized native RAW values and both captures as evidence. If a Ray reference still has only weak-signal rejections after that bounded reacquisition, or the strip cannot establish its three-sample baseline under the same bound, record the target as measurement_unreachable with the failing stage, continue the 12-target protocol, exclude it from numerical Gate-3 aggregates, and force C-A false. During local feedback, reject an unmeasurable candidate without adding it to samples/Jacobian, then try 0.5 and 0.25 backtracks from the current valid command; if all fail, retain the best valid command and run settled final verification. A rejected confirmation is a failed confirmation eligible for thermal rebase; an unmeasurable setup/probe/final verification makes the target measurement_unreachable. Never widen the threshold, censor a frozen feature, extend exposure beyond 1.25x, or downgrade mixed/pixel-count/clipping/nonfinite failures.
25. P4 selected keep_white_lut_diagnostic_only; stop_before_p5: preserve the external white-only 70-point candidate and do not overwrite the default LUT. Stage 5 starts with Ray-only characterization through optimization.measure_ray_hsi_manifold --dry-run; real acquisition requires remote/latest-location, --execute --allow-high-output, and tmp/run-ray-hsi-manifold.zsh. The immutable P6 plan has 73 unique base states and 106 recorded captures: SAT=0 at six primary hues with two repeats plus five total Hue-0 anchors, the 12x6 HSI coarse grid, repeated primary HSI SAT100 endpoints, and two repeats of RGB R/Y/G/C/B/M. Every unique state auto-exposes independently, repeats reuse that state's shutter cache, every command is followed by exact mode readback, and the strip is explicitly turned off before every frame. Analyze a completed run with measurement.ray_hsi_analysis; only frozen-quality accepted native-Bayer features enter comparisons. It reports SAT=0 hue dependence, HSI/RGB endpoint shape after global scale, pooled-zero and periodic hue-dependent-zero affine candidates, radial monotonicity, and Hue closure, but never chooses Gate-5 automatically or implements strip HSI.

Calibration Direction

• White mode should match Ray120c 100% brightness across 1800K..20000K CCT and -1.0..+1.0 G/M offset.
• Use the first-pass CCT/G-M LUT grid from docs/README.md.
• Endpoint detection now uses white-patch CCT/Duv from linear RAW as the search target, then verifies by photographing Ray120c candidates during adaptive CCT/G-M correction. Treat the estimated CCT/Duv as a search heuristic, not a Lightroom-equivalent absolute color temperature.
• White LUT optimization should read the endpoint JSON and interpolate the native white-only G/M between WW and CW. Preserve strong RGB penalties near that native locus, then directionally relax only the channels needed for CCT extension or G/M compensation.
• Neutral patches are the primary objective for white calibration; color patches are auxiliary lower-weight checks.
• Every RGBWW candidate in high-brightness white calibration should keep at least one channel at 4095.
• All optimizer logic should operate on gamma-calibrated channel values, not raw PWM values.
• Color mode should use model-first matching in camera-observed color space plus sparse residual correction where measurements justify it.
• If CCT/G-M calibration measurements reveal a better color-mode model or solver structure, update the strategy to follow the measured evidence.

Safety

• Do not run broad high-output sweeps without an explicit bounded range.
• Confirm channel order, brightness scaling, and thermal behavior before increasing PWM levels.
• All scripts that can leave the RGBWW fixture on must explicitly turn it off before exiting, except dedicated light-on/control tools.