diff --git a/.trellis/spec/hyb-oa/deployment-trialrun.md b/.trellis/spec/hyb-oa/deployment-trialrun.md index 7e5b26a5..37e1aeaa 100644 --- a/.trellis/spec/hyb-oa/deployment-trialrun.md +++ b/.trellis/spec/hyb-oa/deployment-trialrun.md @@ -1425,6 +1425,14 @@ validate_copied_batch(private/batch, source_metadata) Self-cleanup may use `$0` only after validating the fixed basename, regular file type and absence of a symlink; otherwise a non-root outer wrapper owns cleanup of the exact caller-side path. +- A runner's `temporary_access_cleanup=PASS` or equivalent self-report proves + only that the destinations encoded in that runner's cleanup contract are + absent. It is not a zero-residue claim: an installer can place an object + outside that destination set and the runner cannot observe it. Before an + attempt is called clean, an independent post-run audit must enumerate the + installer manifest, runner/probe entries, sudoers candidates, intended and + rejected dependency destinations, user staging, owned runtime and container + identities; every count must be zero and the audit must clean itself. ### 4. Validation & Error Matrix @@ -1468,6 +1476,10 @@ validate_copied_batch(private/batch, source_metadata) - A sudo helper resolves caller staging through root `$HOME`, or cannot prove the fixed identity of `$0` -> leave caller-side cleanup to the non-root outer wrapper and report it explicitly; do not guess or unlink a different home. +- Runner cleanup reports `PASS` but an installer/outer residue count is non-zero + or not independently measured -> the attempt remains non-clean; preserve the + runner self-report as contract-scoped evidence and do not rewrite it as zero + residue. ### 5. Good/Base/Bad Cases @@ -1498,6 +1510,11 @@ validate_copied_batch(private/batch, source_metadata) - Bad: type a site path twice, install helpers below `lib` while the candidate reads `libexec`, or use sudo `$HOME` to delete a script uploaded by the calling user. +- Good: runner cleanup reports its contract destinations clean, then a separate + fixed-target audit independently returns zero for entries, both intended and + previously rejected dependency directories, staging, runtime and containers. +- Bad: summarize `temporary_access_cleanup=PASS` as “all temporary objects are + zero” without checking objects that only the installer knew how to create. ### 6. Tests Required @@ -1549,6 +1566,10 @@ validate_copied_batch(private/batch, source_metadata) regular-file and non-symlink checks may self-clean, while an unverified path must remain owned by a non-root outer cleanup. Assert no root-home path is inferred or emitted. +- Inject an installer destination outside the runner cleanup set. Assert runner + cleanup may truthfully pass for its own targets while the independent audit + reports non-zero and blocks a clean conclusion; after exact cleanup, rerun + the audit and require all enumerated counts plus audit self-cleanup to pass. ### 7. Wrong vs Correct @@ -1573,6 +1594,7 @@ continuous stage transitions -> exact stage sum + one data-only RTO capture only stage/path/exit/category/duration allowlists finalize active stage -> emit fixed diagnostics -> finally teardown raw private runtime sudo helper: validated $0 self-cleanup or exact non-root outer cleanup; never sudo $HOME +runner cleanup PASS + independent fixed-target zero-residue audit -> clean attempt non-zero apply remains FAIL ``` @@ -2783,3 +2805,138 @@ current shared mapping -> fine-grained stages + mutation listeners -> zero sample means dependent checks PAUSED, not auth failure -> reconcile session/audit side effects -> post-window snapshot ``` + +## Scenario: Read-Only Automatic Backup Chain Investigation + +### 1. Scope / Trigger + +- Trigger: the latest off-host archive exceeds its RPO/freshness gate, an + automatic-backup or dead-man service fails, or a restore drill discovers a + stale latest batch. +- Purpose: distinguish scheduler failure, source production failure, interrupted + transfer, finalize failure and alert-delivery behavior without triggering a + backup, changing a timer, reading business contents or modifying an archive. + +### 2. Signatures + +- Production units: `hyb-oa-automatic-backup-daily.service/.timer` and + `hyb-oa-automatic-backup-watchdog.service/.timer`. +- ARM dead-man unit basename is the installed `DEADMAN_SERVICE` contract from + `deploy/trialrun/install-offhost-backup-receiver.sh`; probes must derive or + mechanically compare that value instead of typing a remembered unit name. +- Safe state fields: unit `ActiveState`, `Result`, `ExecMainStatus`, timer last + trigger, effective max-age seconds, summary `status/category/findingCount`, + alert state `status/category`, journal `evaluated sent=true|false` counts, + archive/partial counts and age aggregates, and filesystem percent/free GiB. + +### 3. Contracts + +- A timer being `active` proves scheduling remains enabled; it does not prove + its last service run passed. Always pair timer state/trigger with service + `Result` and exit status. +- Discover the remote account's actual capabilities before installing a probe. + The least-privilege production deployment account is not assumed to have + sudo; a production probe must run unprivileged or stop with an explicit + authority requirement. It must never attempt to widen sudoers itself. +- File absence and access denial are different evidence. Before reporting a + root-owned state file as missing, prove the parent directory is traversable + and readable; otherwise emit `ACCESS_DENIED` and keep the fact unknown. +- Classify receiver inbox objects without outputting names or paths. A partial + with the exact three regular files, valid manifest/run identity and matching + artifact byte counts indicates transfer completion followed by missing or + failed finalize. A partial that fails that contract indicates transfer did + not complete; SHA-256 may remain uncomputed in this diagnostic classification. +- A receiver partial proves that the transfer authenticated and opened its + destination before interruption. Initial private-key rejection, + `authorized_keys` rejection and host-key refusal cannot create that object; + keep investigation on mid-transfer network/session limits, source read, + receiver write and transport I/O. A later SSH-layer disconnect may still + surface as rsync/SSH failure and is not excluded by successful startup. +- For multiple incomplete partials, compare only aggregate byte totals/ratios, + the wrapped rsync exit code and elapsed transfer seconds. Similar stopping + offsets or durations indicate a deterministic limit; materially different + offsets/durations favor intermittent timeout or network interruption. Do not + infer this split from service exit `1`, which is the wrapper's generic code. +- `last-deadman.json` proves the detector decision. `deadman-alert-state.json` + proves the deduplication state. A journal allowlist line with + `evaluated sent=true` proves the helper attempted and accepted one send; it + does not prove a human noticed the message. Repeated `sent=false` for the same + failure is normal deduplication, not a monitoring blind spot. +- Read-only investigation output is an allowlist of status/category/count/age + and capacity fields. Never print unit command lines, raw JSON, raw journal, + remote identities, paths, run ids, webhook data or diagnostic stderr. + +### 4. Validation & Error Matrix + +- Timer inactive or never triggered -> scheduler/configuration branch. +- Timer triggered and service failed before a new local batch is visible -> + source/common-preflight branch; if root state is inaccessible, preserve the + exact phase as unknown instead of guessing. +- New incomplete receiver partial -> `offhost_transfer_incomplete`; do not call + it a finalize failure. +- Partial exists -> initial SSH authentication/host-key acceptance succeeded; + do not spend the first incident cycle rotating keys or rewriting + `authorized_keys` without contrary evidence. +- Receiver partial satisfies the complete byte/manifest contract but no atomic + archive exists -> `offhost_finalize_not_completed`. +- Atomic archive stale + dead-man state stale + `sent=true` -> detector and send + invocation worked; investigate human/channel reception separately. +- Atomic archive stale but dead-man state fresh/missing, service not run, or + timer failed -> monitoring execution blind spot. +- Path cannot be traversed by the approved account -> `ACCESS_DENIED`; request + separate read authority only if the unresolved field materially changes the + decision. + +### 5. Good/Base/Bad Cases + +- Good: daily timer triggered, service failed, ARM has a recent incomplete + partial, dead-man records `archive_stale` and the alert helper records one + `sent=true`; conclude transfer interruption plus successful detector/send + invocation while keeping the production raw phase unknown. +- Base: latest archive is fresh, services pass, inbox has zero partials and the + alert state is healthy. +- Bad: infer “timer active means backup succeeded”, call an inaccessible state + file missing, or describe any `.partial` as finalize failure without checking + whether its three-file byte/manifest contract completed. +- Bad: hard-code a near-match dead-man unit name in a one-off probe and treat an + empty `systemctl show` response as a parser-format problem. + +### 6. Tests Required + +- Extract the receiver dead-man unit basename from the installer and compare it + with every rendered probe before upload; a near-match unit must fail locally. +- Fixture systemd output for active timer/pass service, active timer/failed + service, nonexistent unit and explicit max-age versus script-default max-age. +- Fixture root-owned state directories and assert an unprivileged probe emits + `ACCESS_DENIED`, never `present=no`. +- Fixture incomplete partial, complete-contract partial and archive transaction + temporary entries; assert only aggregate classifications and ages are emitted. +- Fixture two partials with equal and unequal aggregate stopping offsets plus + rsync codes `23/30/12/11/255`; assert the incident summary preserves code and + elapsed-time evidence separately from wrapper exit `1` and does not output + file names. +- Sensitive scan the probe and assert raw `ExecStart`, journal, JSON, paths, + remote identities and run ids cannot appear in stdout. +- Every root probe must self-remove its fixed entry/sudoers and an independent + non-root wrapper must report entry and staging residue separately. + +### 7. Wrong vs Correct + +#### Wrong + +```text +timer active -> backup healthy +state path test returned false -> state missing +partial count > 0 -> finalize failed +``` + +#### Correct + +```text +derive unit identity from installer -> timer trigger + service Result/exit +-> distinguish ACCESS_DENIED from missing +-> partial three-file/manifest/byte classification +-> partial proves initial SSH acceptance; compare byte offset + rsync code + duration +-> dead-man decision + dedup state + sent flag +-> safe scheduler/source/transfer/finalize/alert conclusion +``` diff --git a/.trellis/tasks/08-23-isolated-backup-restore-drill/implement.md b/.trellis/tasks/08-23-isolated-backup-restore-drill/implement.md index f485f6f9..a58aa516 100644 --- a/.trellis/tasks/08-23-isolated-backup-restore-drill/implement.md +++ b/.trellis/tasks/08-23-isolated-backup-restore-drill/implement.md @@ -162,6 +162,18 @@ - [x] ignored/workspace 安装器已把三项依赖安装目标对齐 runner 已批准的 `libexec` 合同；诊断脚本改用经 basename、普通文件和非链接校验的调用路径自清理，不再以 sudo `$HOME` 推断普通用户路径。runner 与候选字节未改、无需重新渲染。 - [ ] 再次实际恢复前重新选择满足 24 小时门禁的新鲜批次，执行只读身份探针并单独申请实际恢复批准；不得复用 Attempt 5 的批次新鲜度或既有实际恢复授权。 +### 08-27 最新批次只读身份探针 + +- [x] 安装器与 rendered runner/probe 的共享路径完成本地静态核对：runner `SELF_PATH/LIB_DIR/SUDOERS_PATH`、probe `SELF_PATH/SUDOERS_PATH` 均在首次特权写入前精确比对；`WORK_ROOT` 未在安装器重复编码，sudoers candidate 从最终路径推导。 +- [x] 探针上传准备中一次 basename 截取错误被远端 SHA-256 校验在普通用户暂存阶段拦截；未执行 sudo、未安装入口、未读取归档。三个已确认截短文件逐个精确清理后，以真实 basename 重传并通过三文件 SHA-256 与 Bash 语法校验。 +- [x] 最新正式批次只读探针执行成功：来源构建与当前生产相同，run id/目录配对、DB/附件产物完整性和 source digest 均通过；探针入口与普通用户暂存残留为 `0/0`。 +- [x] 最新批次年龄 `124137` 秒，约 34 小时 29 分，超过 24 小时门禁 `37737` 秒，约 10 小时 29 分；本次恢复前置判定为 `NO_GO/stale_latest_batch`，不得改选旧批次或执行实际恢复。 +- [x] 用户批准后完成生产与 ARM 两侧只读调查：daily/watchdog timer 均按计划触发但最近 service 均 `exit-code/1`；ARM 最新原子归档超龄且存在两个不完整 partial，定位为生产到 ARM 的传输未完成，不是调度漏跑或完整传输后的 finalize 失败。 +- [x] dead-man 实际阈值为 `93600` 秒并正确记录 `FAIL_OFFHOST_DEADMAN/archive_stale`；journal 聚合存在一次 `sent=true`，证明 detector 与发送调用未被去抖静默抑制。健康 timer 保持 active，两端调查入口和暂存均清理为零残留。 +- [x] 两个 partial 证明 SSH 会话已完成初始认证并开始写入，后续排查不再以密钥失效、authorized_keys 或 host key 初始拒绝为首要假设；独立事件调查优先采集 partial 聚合字节偏移、真实 rsync 退出码和传输持续时间。 +- [ ] 在生产自动备份传输恢复、ARM 形成新的 24 小时内原子批次之前，恢复演练保持阻塞；补跑备份、修复传输配置/网络、修改 timer 或发送测试告警均需新的明确批准。 +- [ ] 将生产到 ARM 的传输中断拆为独立 Trellis 生产事件任务，当前演练任务只保留交叉引用和阻塞关系；创建任务、生产侧只读深查与实际修复均另行取得批准。 + ## 验证命令类别 - 本地/仓库：`bash -n` 覆盖备份、DB 恢复、附件恢复和 receiver 脚本；只读统计 migration 目录。 diff --git a/.trellis/tasks/08-23-isolated-backup-restore-drill/research/restore-drill-evidence.md b/.trellis/tasks/08-23-isolated-backup-restore-drill/research/restore-drill-evidence.md index 6076a6f6..8e7c4c49 100644 --- a/.trellis/tasks/08-23-isolated-backup-restore-drill/research/restore-drill-evidence.md +++ b/.trellis/tasks/08-23-isolated-backup-restore-drill/research/restore-drill-evidence.md @@ -541,3 +541,54 @@ runner 随后在 `entry_contract` 阶段以 `restore_library_missing` 失败关 本地完整复核结果：影子恢复 runner 合同全部 `PASS`，真实 PostgreSQL 16 fixture 的 Docker stdin/host-file 路径继续通过，合成成对备份/恢复演练 `PASS`，自动异机备份合同 `74/74 PASS`；task validate、Trellis governance、process baseline、敏感扫描和 `git diff --check` 通过。文档审计错误 `0`，只保留既有未登记文件 warning；`scripts/restore-db.sh` 相对生产构建字节漂移 `0`，本地测试容器残留 `0`。 任何再次实际恢复仍须重新选择满足 24 小时门禁的新鲜批次、重新执行只读身份探针并取得独立批准。本节只闭环 Attempt 5 的环境清理和安装器路径缺陷，不回答最新正式备份是否可恢复。 + +## 08-27 最新正式批次只读身份探针 + +日期：2026-08-27 + +本节只追加用户批准的最新正式批次只读探针。未复制、解包或恢复正式归档，未启动 Docker，未修改腾讯云生产、ARM 正式归档、timer 或企业微信。 + +探针准备前完成了安装器共享路径静态核对：runner 的 `SELF_PATH`、`LIB_DIR`、`SUDOERS_PATH` 与安装器目标精确一致，probe 的 `SELF_PATH`、`SUDOERS_PATH` 与探针安装器精确一致；`WORK_ROOT` 只由 runner 持有，安装器没有第二份编码；sudoers candidate 由最终 sudoers 路径推导。上述比较均位于首次 `sudo install` 之前，失败即不产生特权写入。 + +第一次上传准备因本地编排器错误截取 basename，三个文件以截短名称进入普通用户暂存；远端 SHA-256 门禁立即失败。该失败发生在 sudo 和探针执行之前，未安装 root 入口、未读取归档。确认暂存目录恰含三个预期的截短普通文件后，逐个精确删除并移除本轮暂存和安装器，再使用真实 basename 重新上传；重传包的两项 SHA-256、Bash 语法和三文件集合均通过。该 finding 属于 `local_probe_upload_orchestration_defect`，未影响 probe/runner 字节或 ARM 正式状态。 + +现场探针结果： + +| 检查项 | 结果 | +| --- | --- | +| 探针执行 | `PASS` / exit `0` | +| 最新批次构建短标识 | `7b86ffb3`，与当前生产相同 | +| 最新批次年龄 | `124137` 秒，约 34 小时 29 分 | +| 24 小时门禁 | `FAIL`；超出 `37737` 秒，约 10 小时 29 分 | +| run id 与目录配对 | `PASS` | +| DB/附件大小与 SHA-256 | `PASS` | +| source image digest | 存在 | +| manifest 脱敏摘要前缀 | `ede8a77d9912` | +| 探针合同内临时访问清理 | `PASS` | +| 探针入口/普通用户暂存残留 | `0/0` | + +`probe_result=PASS` 只表示只读探针完整执行并成功校验所见批次，不等于恢复前置门禁通过。按 PRD 的 24 小时硬门禁，本次结论为 `NO_GO/stale_latest_batch`：不得执行实际恢复，也不得改选更旧批次掩盖最新恢复点不新鲜。构建、身份和产物完整性通过说明当前证据不指向归档损坏；但仅凭本次探针不能区分源端自动备份未产出、传输/finalize 失败或调度时点变化。 + +下一步须另行批准只读调查生产自动备份最近运行状态、ARM receiver 最新原子归档时刻以及 dead-man/健康 timer 状态。该调查不得手工触发备份或修改 timer。只有新的正式批次满足 24 小时门禁并重新通过身份探针后，才能再次申请实际隔离恢复批准；既有恢复授权不延续。 + +## 08-27 自动备份链与 dead-man 只读调查 + +日期：2026-08-27 + +用户批准的调查只读取生产自动备份 systemd 聚合状态、ARM receiver 原子归档/inbox 聚合状态、dead-man 状态和两端文件系统容量。未触发备份、未修改 timer、未执行恢复、未改正式归档、未发送测试告警，也未输出远端身份、现场路径、原始 JSON、journal、凭据或完整 run id。 + +生产侧结果：daily timer 与 watchdog timer 均为 `active`，最近分别在当日计划时点触发；对应 service 最近一次均为 `failed/result=exit-code/exit=1`。因此“timer 未调度”被排除，失败位于两项服务共用的备份运行合同或其后续路径。批准的部署账号不具备 sudo，也不能穿越 root-owned 状态/本地备份目录；调查明确记录 `ACCESS_DENIED`，没有把无权读取改写成状态文件或本地批次不存在。该权限边界使生产侧原始失败 phase 保持未知，不据此猜测具体 stderr。 + +ARM 侧结果：dead-man timer 与公网健康 timer 均为 `active`；dead-man service 最近一次按失败状态退出。实际 dead-man 阈值为显式参数 `93600` 秒（26 小时）。`last-deadman` 为 `FAIL_OFFHOST_DEADMAN/archive_stale/findingCount=1`，alert state 同为 `FAIL_OFFHOST_DEADMAN/archive_stale`。最近 72 小时 journal 仅做 allowlist 聚合，得到 `sent=true` 一次、`sent=false` 两次，最近一次为 `sent=true`。该次 dead-man timer 触发锚点为 2026-08-27 06:30:06（Asia/Shanghai），alert state 在同轮随即更新，可作为本次发送调用时刻。恢复演练探针在同日稍后首先促成人工响应，但当时未独立采集“人员实际知晓”的精确墙钟时刻，因此本轮不能伪造精确响应延迟，只能登记为“发送调用早于人工响应、精确差值未测”。后续事件任务必须同时记录 `sent=true` 与首次确认时刻，直接计算真实响应延迟。 + +检测侧结论为正面：真实超龄被正确检测，告警 helper 至少完成了一次发送调用，且相同失败的 `sent=false` 属于正常去抖；08-23 去抖改造没有造成检测或发送调用盲区。响应侧仍有 finding：本次备份中断最终由恢复演练探针触发处置，而不是由既有告警形成及时响应。证据不证明具体人员已阅读消息，后续应调查群消息送达、可见性、责任人确认和升级流程，不能把检测调用成功等同于闭环响应。 + +receiver 聚合结果：正式原子日归档共 7 个，最新归档在复核时年龄约 43 小时 40 分；inbox 有 2 个 `.partial`，正式归档事务临时对象为 0、inbox 其他对象为 0。两个 partial 均不满足“三个普通文件 + manifest/run 身份 + 产物字节数”的完整传输合同，完整合同数量为 0、不完整数量为 2；最新 partial 年龄约 19 小时 40 分，最旧 partial 约 31 天 10 小时。由此将当前事故分类为 `offhost_transfer_incomplete`：生产 daily 确实触发并在 ARM 留下未完成传输，不能归为源端完全未产出、timer 漏调度或“传输已经完整但 finalize 失败”。partial 的存在同时证明 rsync 已通过初始 SSH 认证、host key 校验和 receiver 授权并开始写入，因此密钥失效、authorized_keys 初始拒绝和 host key 初始拒绝不是本次首要候选；排查应收缩到传输中段的超时/网络中断、源端读取、目标端写入或会话/传输限制。最旧 partial 同时是历史清理/观测 finding，但本轮没有删除它。 + +当前只读字段还不足以在上述传输中段候选之间继续裁决。下一轮独立生产事件调查应比较两个 partial 的聚合字节大小/停止比例，读取 wrapper 私有诊断中的真实 rsync 退出码，并记录每轮传输持续时间：相近停止偏移或相近时长支持确定性限制，不同偏移/时长更支持网络或超时；service 的通用 exit `1` 不得替代 rsync 原始码。该调查仍只输出字节聚合、退出码和耗时，不输出文件名、路径或 stderr 原文。 + +容量必须与传输修复并行处理：生产侧相关路径位于同一可识别文件系统，使用率约 82%、可用约 10 GiB；ARM archive/inbox 文件系统使用率约 53%、可用约 89 GiB。现有证据仍不足以把这次 rsync 中断直接归因于磁盘不足，但自动备份会先在生产生成完整 dump 与 storage 归档，持续失败还会反复生成/保留本地产物；10 GiB 对该流程属于薄余量。若继续下降，故障可能从“传输中断”升级为“源端无法产出”，因此容量治理不是低优先级后续项，而是独立生产事件中的并行处置项。 + +调查器本身出现并安全关闭了两类本地缺陷：首次生产探针错误假设部署账号可 sudo，安装前即被权限拒绝；前两次 ARM 探针把 dead-man unit 写成近似但不存在的名称，均在状态读取前以合同不可读失败。每次 root 入口、sudoers candidate、普通用户暂存均为 0。最终版本从仓库 receiver 安装器机械提取并比较 unit basename，并将生产探针改为无提权只读执行；Bash/Python 语法、两种 ExecStart 解析 fixture 和敏感扫描通过。上述缺陷不改变生产、ARM、归档、timer 或告警状态，也没有伪造调查结果。 + +当前独立生产 finding 为：最新可用异机恢复点已超过 24 小时目标，自动备份传输链正在失败；恢复演练因此保持阻塞。该事件不再作为恢复演练的普通 finding 排队处理，建议另建 Trellis 生产事件任务承载传输/容量调查与修复，当前演练任务仅保留证据交叉引用和阻塞关系。后续优先顺序是另行批准修复生产到 ARM 的传输链并形成新的原子批次，确认自动备份/两端 dead-man 与 watchdog 恢复，再对新批次重新执行身份探针。手工补跑备份、修改网络/SSH/timer、清理 partial 或发送测试告警均不在本次只读批准内。