记录(08-23-isolated-backup-restore-drill): 固化备份传输中断调查

.trellis/spec/hyb-oa/deployment-trialrun.md

@@ -1425,6 +1425,14 @@ validate_copied_batch(private/batch, source_metadata)
   Self-cleanup may use `$0` only after validating the fixed basename, regular
   file type and absence of a symlink; otherwise a non-root outer wrapper owns
   cleanup of the exact caller-side path.
+- A runner's `temporary_access_cleanup=PASS` or equivalent self-report proves
+  only that the destinations encoded in that runner's cleanup contract are
+  absent. It is not a zero-residue claim: an installer can place an object
+  outside that destination set and the runner cannot observe it. Before an
+  attempt is called clean, an independent post-run audit must enumerate the
+  installer manifest, runner/probe entries, sudoers candidates, intended and
+  rejected dependency destinations, user staging, owned runtime and container
+  identities; every count must be zero and the audit must clean itself.
 
 ### 4. Validation & Error Matrix
 
@@ -1468,6 +1476,10 @@ validate_copied_batch(private/batch, source_metadata)
 - A sudo helper resolves caller staging through root `$HOME`, or cannot prove
   the fixed identity of `$0` -> leave caller-side cleanup to the non-root outer
   wrapper and report it explicitly; do not guess or unlink a different home.
+- Runner cleanup reports `PASS` but an installer/outer residue count is non-zero
+  or not independently measured -> the attempt remains non-clean; preserve the
+  runner self-report as contract-scoped evidence and do not rewrite it as zero
+  residue.
 
 ### 5. Good/Base/Bad Cases
 
@@ -1498,6 +1510,11 @@ validate_copied_batch(private/batch, source_metadata)
 - Bad: type a site path twice, install helpers below `lib` while the candidate
   reads `libexec`, or use sudo `$HOME` to delete a script uploaded by the
   calling user.
+- Good: runner cleanup reports its contract destinations clean, then a separate
+  fixed-target audit independently returns zero for entries, both intended and
+  previously rejected dependency directories, staging, runtime and containers.
+- Bad: summarize `temporary_access_cleanup=PASS` as “all temporary objects are
+  zero” without checking objects that only the installer knew how to create.
 
 ### 6. Tests Required
 
@@ -1549,6 +1566,10 @@ validate_copied_batch(private/batch, source_metadata)
   regular-file and non-symlink checks may self-clean, while an unverified path
   must remain owned by a non-root outer cleanup. Assert no root-home path is
   inferred or emitted.
+- Inject an installer destination outside the runner cleanup set. Assert runner
+  cleanup may truthfully pass for its own targets while the independent audit
+  reports non-zero and blocks a clean conclusion; after exact cleanup, rerun
+  the audit and require all enumerated counts plus audit self-cleanup to pass.
 
 ### 7. Wrong vs Correct
 
@@ -1573,6 +1594,7 @@ continuous stage transitions -> exact stage sum + one data-only RTO
 capture only stage/path/exit/category/duration allowlists
 finalize active stage -> emit fixed diagnostics -> finally teardown raw private runtime
 sudo helper: validated $0 self-cleanup or exact non-root outer cleanup; never sudo $HOME
+runner cleanup PASS + independent fixed-target zero-residue audit -> clean attempt
 non-zero apply remains FAIL
 ```
 
@@ -2783,3 +2805,138 @@ current shared mapping -> fine-grained stages + mutation listeners
 -> zero sample means dependent checks PAUSED, not auth failure
 -> reconcile session/audit side effects -> post-window snapshot
 ```
+
+## Scenario: Read-Only Automatic Backup Chain Investigation
+
+### 1. Scope / Trigger
+
+- Trigger: the latest off-host archive exceeds its RPO/freshness gate, an
+  automatic-backup or dead-man service fails, or a restore drill discovers a
+  stale latest batch.
+- Purpose: distinguish scheduler failure, source production failure, interrupted
+  transfer, finalize failure and alert-delivery behavior without triggering a
+  backup, changing a timer, reading business contents or modifying an archive.
+
+### 2. Signatures
+
+- Production units: `hyb-oa-automatic-backup-daily.service/.timer` and
+  `hyb-oa-automatic-backup-watchdog.service/.timer`.
+- ARM dead-man unit basename is the installed `DEADMAN_SERVICE` contract from
+  `deploy/trialrun/install-offhost-backup-receiver.sh`; probes must derive or
+  mechanically compare that value instead of typing a remembered unit name.
+- Safe state fields: unit `ActiveState`, `Result`, `ExecMainStatus`, timer last
+  trigger, effective max-age seconds, summary `status/category/findingCount`,
+  alert state `status/category`, journal `evaluated sent=true|false` counts,
+  archive/partial counts and age aggregates, and filesystem percent/free GiB.
+
+### 3. Contracts
+
+- A timer being `active` proves scheduling remains enabled; it does not prove
+  its last service run passed. Always pair timer state/trigger with service
+  `Result` and exit status.
+- Discover the remote account's actual capabilities before installing a probe.
+  The least-privilege production deployment account is not assumed to have
+  sudo; a production probe must run unprivileged or stop with an explicit
+  authority requirement. It must never attempt to widen sudoers itself.
+- File absence and access denial are different evidence. Before reporting a
+  root-owned state file as missing, prove the parent directory is traversable
+  and readable; otherwise emit `ACCESS_DENIED` and keep the fact unknown.
+- Classify receiver inbox objects without outputting names or paths. A partial
+  with the exact three regular files, valid manifest/run identity and matching
+  artifact byte counts indicates transfer completion followed by missing or
+  failed finalize. A partial that fails that contract indicates transfer did
+  not complete; SHA-256 may remain uncomputed in this diagnostic classification.
+- A receiver partial proves that the transfer authenticated and opened its
+  destination before interruption. Initial private-key rejection,
+  `authorized_keys` rejection and host-key refusal cannot create that object;
+  keep investigation on mid-transfer network/session limits, source read,
+  receiver write and transport I/O. A later SSH-layer disconnect may still
+  surface as rsync/SSH failure and is not excluded by successful startup.
+- For multiple incomplete partials, compare only aggregate byte totals/ratios,
+  the wrapped rsync exit code and elapsed transfer seconds. Similar stopping
+  offsets or durations indicate a deterministic limit; materially different
+  offsets/durations favor intermittent timeout or network interruption. Do not
+  infer this split from service exit `1`, which is the wrapper's generic code.
+- `last-deadman.json` proves the detector decision. `deadman-alert-state.json`
+  proves the deduplication state. A journal allowlist line with
+  `evaluated sent=true` proves the helper attempted and accepted one send; it
+  does not prove a human noticed the message. Repeated `sent=false` for the same
+  failure is normal deduplication, not a monitoring blind spot.
+- Read-only investigation output is an allowlist of status/category/count/age
+  and capacity fields. Never print unit command lines, raw JSON, raw journal,
+  remote identities, paths, run ids, webhook data or diagnostic stderr.
+
+### 4. Validation & Error Matrix
+
+- Timer inactive or never triggered -> scheduler/configuration branch.
+- Timer triggered and service failed before a new local batch is visible ->
+  source/common-preflight branch; if root state is inaccessible, preserve the
+  exact phase as unknown instead of guessing.
+- New incomplete receiver partial -> `offhost_transfer_incomplete`; do not call
+  it a finalize failure.
+- Partial exists -> initial SSH authentication/host-key acceptance succeeded;
+  do not spend the first incident cycle rotating keys or rewriting
+  `authorized_keys` without contrary evidence.
+- Receiver partial satisfies the complete byte/manifest contract but no atomic
+  archive exists -> `offhost_finalize_not_completed`.
+- Atomic archive stale + dead-man state stale + `sent=true` -> detector and send
+  invocation worked; investigate human/channel reception separately.
+- Atomic archive stale but dead-man state fresh/missing, service not run, or
+  timer failed -> monitoring execution blind spot.
+- Path cannot be traversed by the approved account -> `ACCESS_DENIED`; request
+  separate read authority only if the unresolved field materially changes the
+  decision.
+
+### 5. Good/Base/Bad Cases
+
+- Good: daily timer triggered, service failed, ARM has a recent incomplete
+  partial, dead-man records `archive_stale` and the alert helper records one
+  `sent=true`; conclude transfer interruption plus successful detector/send
+  invocation while keeping the production raw phase unknown.
+- Base: latest archive is fresh, services pass, inbox has zero partials and the
+  alert state is healthy.
+- Bad: infer “timer active means backup succeeded”, call an inaccessible state
+  file missing, or describe any `.partial` as finalize failure without checking
+  whether its three-file byte/manifest contract completed.
+- Bad: hard-code a near-match dead-man unit name in a one-off probe and treat an
+  empty `systemctl show` response as a parser-format problem.
+
+### 6. Tests Required
+
+- Extract the receiver dead-man unit basename from the installer and compare it
+  with every rendered probe before upload; a near-match unit must fail locally.
+- Fixture systemd output for active timer/pass service, active timer/failed
+  service, nonexistent unit and explicit max-age versus script-default max-age.
+- Fixture root-owned state directories and assert an unprivileged probe emits
+  `ACCESS_DENIED`, never `present=no`.
+- Fixture incomplete partial, complete-contract partial and archive transaction
+  temporary entries; assert only aggregate classifications and ages are emitted.
+- Fixture two partials with equal and unequal aggregate stopping offsets plus
+  rsync codes `23/30/12/11/255`; assert the incident summary preserves code and
+  elapsed-time evidence separately from wrapper exit `1` and does not output
+  file names.
+- Sensitive scan the probe and assert raw `ExecStart`, journal, JSON, paths,
+  remote identities and run ids cannot appear in stdout.
+- Every root probe must self-remove its fixed entry/sudoers and an independent
+  non-root wrapper must report entry and staging residue separately.
+
+### 7. Wrong vs Correct
+
+#### Wrong
+
+```text
+timer active -> backup healthy
+state path test returned false -> state missing
+partial count > 0 -> finalize failed
+```
+
+#### Correct
+
+```text
+derive unit identity from installer -> timer trigger + service Result/exit
+-> distinguish ACCESS_DENIED from missing
+-> partial three-file/manifest/byte classification
+-> partial proves initial SSH acceptance; compare byte offset + rsync code + duration
+-> dead-man decision + dedup state + sent flag
+-> safe scheduler/source/transfer/finalize/alert conclusion
+```