记录(08-23-isolated-backup-restore-drill): 固化备份传输中断调查

这个提交包含在:
2026-08-27 23:25:33 +08:00
父节点 f84e6cf4e3
当前提交 a1c59e2a78
共修改 3 个文件,包含 220 行新增和 0 行删除
+157
查看文件
@@ -1425,6 +1425,14 @@ validate_copied_batch(private/batch, source_metadata)
Self-cleanup may use `$0` only after validating the fixed basename, regular
file type and absence of a symlink; otherwise a non-root outer wrapper owns
cleanup of the exact caller-side path.
- A runner's `temporary_access_cleanup=PASS` or equivalent self-report proves
only that the destinations encoded in that runner's cleanup contract are
absent. It is not a zero-residue claim: an installer can place an object
outside that destination set and the runner cannot observe it. Before an
attempt is called clean, an independent post-run audit must enumerate the
installer manifest, runner/probe entries, sudoers candidates, intended and
rejected dependency destinations, user staging, owned runtime and container
identities; every count must be zero and the audit must clean itself.
### 4. Validation & Error Matrix
@@ -1468,6 +1476,10 @@ validate_copied_batch(private/batch, source_metadata)
- A sudo helper resolves caller staging through root `$HOME`, or cannot prove
the fixed identity of `$0` -> leave caller-side cleanup to the non-root outer
wrapper and report it explicitly; do not guess or unlink a different home.
- Runner cleanup reports `PASS` but an installer/outer residue count is non-zero
or not independently measured -> the attempt remains non-clean; preserve the
runner self-report as contract-scoped evidence and do not rewrite it as zero
residue.
### 5. Good/Base/Bad Cases
@@ -1498,6 +1510,11 @@ validate_copied_batch(private/batch, source_metadata)
- Bad: type a site path twice, install helpers below `lib` while the candidate
reads `libexec`, or use sudo `$HOME` to delete a script uploaded by the
calling user.
- Good: runner cleanup reports its contract destinations clean, then a separate
fixed-target audit independently returns zero for entries, both intended and
previously rejected dependency directories, staging, runtime and containers.
- Bad: summarize `temporary_access_cleanup=PASS` as “all temporary objects are
zero” without checking objects that only the installer knew how to create.
### 6. Tests Required
@@ -1549,6 +1566,10 @@ validate_copied_batch(private/batch, source_metadata)
regular-file and non-symlink checks may self-clean, while an unverified path
must remain owned by a non-root outer cleanup. Assert no root-home path is
inferred or emitted.
- Inject an installer destination outside the runner cleanup set. Assert runner
cleanup may truthfully pass for its own targets while the independent audit
reports non-zero and blocks a clean conclusion; after exact cleanup, rerun
the audit and require all enumerated counts plus audit self-cleanup to pass.
### 7. Wrong vs Correct
@@ -1573,6 +1594,7 @@ continuous stage transitions -> exact stage sum + one data-only RTO
capture only stage/path/exit/category/duration allowlists
finalize active stage -> emit fixed diagnostics -> finally teardown raw private runtime
sudo helper: validated $0 self-cleanup or exact non-root outer cleanup; never sudo $HOME
runner cleanup PASS + independent fixed-target zero-residue audit -> clean attempt
non-zero apply remains FAIL
```
@@ -2783,3 +2805,138 @@ current shared mapping -> fine-grained stages + mutation listeners
-> zero sample means dependent checks PAUSED, not auth failure
-> reconcile session/audit side effects -> post-window snapshot
```
## Scenario: Read-Only Automatic Backup Chain Investigation
### 1. Scope / Trigger
- Trigger: the latest off-host archive exceeds its RPO/freshness gate, an
automatic-backup or dead-man service fails, or a restore drill discovers a
stale latest batch.
- Purpose: distinguish scheduler failure, source production failure, interrupted
transfer, finalize failure and alert-delivery behavior without triggering a
backup, changing a timer, reading business contents or modifying an archive.
### 2. Signatures
- Production units: `hyb-oa-automatic-backup-daily.service/.timer` and
`hyb-oa-automatic-backup-watchdog.service/.timer`.
- ARM dead-man unit basename is the installed `DEADMAN_SERVICE` contract from
`deploy/trialrun/install-offhost-backup-receiver.sh`; probes must derive or
mechanically compare that value instead of typing a remembered unit name.
- Safe state fields: unit `ActiveState`, `Result`, `ExecMainStatus`, timer last
trigger, effective max-age seconds, summary `status/category/findingCount`,
alert state `status/category`, journal `evaluated sent=true|false` counts,
archive/partial counts and age aggregates, and filesystem percent/free GiB.
### 3. Contracts
- A timer being `active` proves scheduling remains enabled; it does not prove
its last service run passed. Always pair timer state/trigger with service
`Result` and exit status.
- Discover the remote account's actual capabilities before installing a probe.
The least-privilege production deployment account is not assumed to have
sudo; a production probe must run unprivileged or stop with an explicit
authority requirement. It must never attempt to widen sudoers itself.
- File absence and access denial are different evidence. Before reporting a
root-owned state file as missing, prove the parent directory is traversable
and readable; otherwise emit `ACCESS_DENIED` and keep the fact unknown.
- Classify receiver inbox objects without outputting names or paths. A partial
with the exact three regular files, valid manifest/run identity and matching
artifact byte counts indicates transfer completion followed by missing or
failed finalize. A partial that fails that contract indicates transfer did
not complete; SHA-256 may remain uncomputed in this diagnostic classification.
- A receiver partial proves that the transfer authenticated and opened its
destination before interruption. Initial private-key rejection,
`authorized_keys` rejection and host-key refusal cannot create that object;
keep investigation on mid-transfer network/session limits, source read,
receiver write and transport I/O. A later SSH-layer disconnect may still
surface as rsync/SSH failure and is not excluded by successful startup.
- For multiple incomplete partials, compare only aggregate byte totals/ratios,
the wrapped rsync exit code and elapsed transfer seconds. Similar stopping
offsets or durations indicate a deterministic limit; materially different
offsets/durations favor intermittent timeout or network interruption. Do not
infer this split from service exit `1`, which is the wrapper's generic code.
- `last-deadman.json` proves the detector decision. `deadman-alert-state.json`
proves the deduplication state. A journal allowlist line with
`evaluated sent=true` proves the helper attempted and accepted one send; it
does not prove a human noticed the message. Repeated `sent=false` for the same
failure is normal deduplication, not a monitoring blind spot.
- Read-only investigation output is an allowlist of status/category/count/age
and capacity fields. Never print unit command lines, raw JSON, raw journal,
remote identities, paths, run ids, webhook data or diagnostic stderr.
### 4. Validation & Error Matrix
- Timer inactive or never triggered -> scheduler/configuration branch.
- Timer triggered and service failed before a new local batch is visible ->
source/common-preflight branch; if root state is inaccessible, preserve the
exact phase as unknown instead of guessing.
- New incomplete receiver partial -> `offhost_transfer_incomplete`; do not call
it a finalize failure.
- Partial exists -> initial SSH authentication/host-key acceptance succeeded;
do not spend the first incident cycle rotating keys or rewriting
`authorized_keys` without contrary evidence.
- Receiver partial satisfies the complete byte/manifest contract but no atomic
archive exists -> `offhost_finalize_not_completed`.
- Atomic archive stale + dead-man state stale + `sent=true` -> detector and send
invocation worked; investigate human/channel reception separately.
- Atomic archive stale but dead-man state fresh/missing, service not run, or
timer failed -> monitoring execution blind spot.
- Path cannot be traversed by the approved account -> `ACCESS_DENIED`; request
separate read authority only if the unresolved field materially changes the
decision.
### 5. Good/Base/Bad Cases
- Good: daily timer triggered, service failed, ARM has a recent incomplete
partial, dead-man records `archive_stale` and the alert helper records one
`sent=true`; conclude transfer interruption plus successful detector/send
invocation while keeping the production raw phase unknown.
- Base: latest archive is fresh, services pass, inbox has zero partials and the
alert state is healthy.
- Bad: infer “timer active means backup succeeded”, call an inaccessible state
file missing, or describe any `.partial` as finalize failure without checking
whether its three-file byte/manifest contract completed.
- Bad: hard-code a near-match dead-man unit name in a one-off probe and treat an
empty `systemctl show` response as a parser-format problem.
### 6. Tests Required
- Extract the receiver dead-man unit basename from the installer and compare it
with every rendered probe before upload; a near-match unit must fail locally.
- Fixture systemd output for active timer/pass service, active timer/failed
service, nonexistent unit and explicit max-age versus script-default max-age.
- Fixture root-owned state directories and assert an unprivileged probe emits
`ACCESS_DENIED`, never `present=no`.
- Fixture incomplete partial, complete-contract partial and archive transaction
temporary entries; assert only aggregate classifications and ages are emitted.
- Fixture two partials with equal and unequal aggregate stopping offsets plus
rsync codes `23/30/12/11/255`; assert the incident summary preserves code and
elapsed-time evidence separately from wrapper exit `1` and does not output
file names.
- Sensitive scan the probe and assert raw `ExecStart`, journal, JSON, paths,
remote identities and run ids cannot appear in stdout.
- Every root probe must self-remove its fixed entry/sudoers and an independent
non-root wrapper must report entry and staging residue separately.
### 7. Wrong vs Correct
#### Wrong
```text
timer active -> backup healthy
state path test returned false -> state missing
partial count > 0 -> finalize failed
```
#### Correct
```text
derive unit identity from installer -> timer trigger + service Result/exit
-> distinguish ACCESS_DENIED from missing
-> partial three-file/manifest/byte classification
-> partial proves initial SSH acceptance; compare byte offset + rsync code + duration
-> dead-man decision + dedup state + sent flag
-> safe scheduler/source/transfer/finalize/alert conclusion
```
@@ -162,6 +162,18 @@
- [x] ignored/workspace 安装器已把三项依赖安装目标对齐 runner 已批准的 `libexec` 合同;诊断脚本改用经 basename、普通文件和非链接校验的调用路径自清理,不再以 sudo `$HOME` 推断普通用户路径。runner 与候选字节未改、无需重新渲染。
- [ ] 再次实际恢复前重新选择满足 24 小时门禁的新鲜批次,执行只读身份探针并单独申请实际恢复批准;不得复用 Attempt 5 的批次新鲜度或既有实际恢复授权。
### 08-27 最新批次只读身份探针
- [x] 安装器与 rendered runner/probe 的共享路径完成本地静态核对:runner `SELF_PATH/LIB_DIR/SUDOERS_PATH`、probe `SELF_PATH/SUDOERS_PATH` 均在首次特权写入前精确比对;`WORK_ROOT` 未在安装器重复编码,sudoers candidate 从最终路径推导。
- [x] 探针上传准备中一次 basename 截取错误被远端 SHA-256 校验在普通用户暂存阶段拦截;未执行 sudo、未安装入口、未读取归档。三个已确认截短文件逐个精确清理后,以真实 basename 重传并通过三文件 SHA-256 与 Bash 语法校验。
- [x] 最新正式批次只读探针执行成功:来源构建与当前生产相同,run id/目录配对、DB/附件产物完整性和 source digest 均通过;探针入口与普通用户暂存残留为 `0/0`。
- [x] 最新批次年龄 `124137` 秒,约 34 小时 29 分,超过 24 小时门禁 `37737` 秒,约 10 小时 29 分;本次恢复前置判定为 `NO_GO/stale_latest_batch`,不得改选旧批次或执行实际恢复。
- [x] 用户批准后完成生产与 ARM 两侧只读调查:daily/watchdog timer 均按计划触发但最近 service 均 `exit-code/1`;ARM 最新原子归档超龄且存在两个不完整 partial,定位为生产到 ARM 的传输未完成,不是调度漏跑或完整传输后的 finalize 失败。
- [x] dead-man 实际阈值为 `93600` 秒并正确记录 `FAIL_OFFHOST_DEADMAN/archive_stale`;journal 聚合存在一次 `sent=true`,证明 detector 与发送调用未被去抖静默抑制。健康 timer 保持 active,两端调查入口和暂存均清理为零残留。
- [x] 两个 partial 证明 SSH 会话已完成初始认证并开始写入,后续排查不再以密钥失效、authorized_keys 或 host key 初始拒绝为首要假设;独立事件调查优先采集 partial 聚合字节偏移、真实 rsync 退出码和传输持续时间。
- [ ] 在生产自动备份传输恢复、ARM 形成新的 24 小时内原子批次之前,恢复演练保持阻塞;补跑备份、修复传输配置/网络、修改 timer 或发送测试告警均需新的明确批准。
- [ ] 将生产到 ARM 的传输中断拆为独立 Trellis 生产事件任务,当前演练任务只保留交叉引用和阻塞关系;创建任务、生产侧只读深查与实际修复均另行取得批准。
## 验证命令类别
- 本地/仓库:`bash -n` 覆盖备份、DB 恢复、附件恢复和 receiver 脚本;只读统计 migration 目录。
@@ -541,3 +541,54 @@ runner 随后在 `entry_contract` 阶段以 `restore_library_missing` 失败关
本地完整复核结果:影子恢复 runner 合同全部 `PASS`,真实 PostgreSQL 16 fixture 的 Docker stdin/host-file 路径继续通过,合成成对备份/恢复演练 `PASS`,自动异机备份合同 `74/74 PASS`;task validate、Trellis governance、process baseline、敏感扫描和 `git diff --check` 通过。文档审计错误 `0`,只保留既有未登记文件 warning;`scripts/restore-db.sh` 相对生产构建字节漂移 `0`,本地测试容器残留 `0`。
任何再次实际恢复仍须重新选择满足 24 小时门禁的新鲜批次、重新执行只读身份探针并取得独立批准。本节只闭环 Attempt 5 的环境清理和安装器路径缺陷,不回答最新正式备份是否可恢复。
## 08-27 最新正式批次只读身份探针
日期:2026-08-27
本节只追加用户批准的最新正式批次只读探针。未复制、解包或恢复正式归档,未启动 Docker,未修改腾讯云生产、ARM 正式归档、timer 或企业微信。
探针准备前完成了安装器共享路径静态核对:runner 的 `SELF_PATH`、`LIB_DIR`、`SUDOERS_PATH` 与安装器目标精确一致,probe 的 `SELF_PATH`、`SUDOERS_PATH` 与探针安装器精确一致;`WORK_ROOT` 只由 runner 持有,安装器没有第二份编码;sudoers candidate 由最终 sudoers 路径推导。上述比较均位于首次 `sudo install` 之前,失败即不产生特权写入。
第一次上传准备因本地编排器错误截取 basename,三个文件以截短名称进入普通用户暂存;远端 SHA-256 门禁立即失败。该失败发生在 sudo 和探针执行之前,未安装 root 入口、未读取归档。确认暂存目录恰含三个预期的截短普通文件后,逐个精确删除并移除本轮暂存和安装器,再使用真实 basename 重新上传;重传包的两项 SHA-256、Bash 语法和三文件集合均通过。该 finding 属于 `local_probe_upload_orchestration_defect`,未影响 probe/runner 字节或 ARM 正式状态。
现场探针结果:
| 检查项 | 结果 |
| --- | --- |
| 探针执行 | `PASS` / exit `0` |
| 最新批次构建短标识 | `7b86ffb3`,与当前生产相同 |
| 最新批次年龄 | `124137` 秒,约 34 小时 29 分 |
| 24 小时门禁 | `FAIL`;超出 `37737` 秒,约 10 小时 29 分 |
| run id 与目录配对 | `PASS` |
| DB/附件大小与 SHA-256 | `PASS` |
| source image digest | 存在 |
| manifest 脱敏摘要前缀 | `ede8a77d9912` |
| 探针合同内临时访问清理 | `PASS` |
| 探针入口/普通用户暂存残留 | `0/0` |
`probe_result=PASS` 只表示只读探针完整执行并成功校验所见批次,不等于恢复前置门禁通过。按 PRD 的 24 小时硬门禁,本次结论为 `NO_GO/stale_latest_batch`:不得执行实际恢复,也不得改选更旧批次掩盖最新恢复点不新鲜。构建、身份和产物完整性通过说明当前证据不指向归档损坏;但仅凭本次探针不能区分源端自动备份未产出、传输/finalize 失败或调度时点变化。
下一步须另行批准只读调查生产自动备份最近运行状态、ARM receiver 最新原子归档时刻以及 dead-man/健康 timer 状态。该调查不得手工触发备份或修改 timer。只有新的正式批次满足 24 小时门禁并重新通过身份探针后,才能再次申请实际隔离恢复批准;既有恢复授权不延续。
## 08-27 自动备份链与 dead-man 只读调查
日期:2026-08-27
用户批准的调查只读取生产自动备份 systemd 聚合状态、ARM receiver 原子归档/inbox 聚合状态、dead-man 状态和两端文件系统容量。未触发备份、未修改 timer、未执行恢复、未改正式归档、未发送测试告警,也未输出远端身份、现场路径、原始 JSON、journal、凭据或完整 run id。
生产侧结果:daily timer 与 watchdog timer 均为 `active`,最近分别在当日计划时点触发;对应 service 最近一次均为 `failed/result=exit-code/exit=1`。因此“timer 未调度”被排除,失败位于两项服务共用的备份运行合同或其后续路径。批准的部署账号不具备 sudo,也不能穿越 root-owned 状态/本地备份目录;调查明确记录 `ACCESS_DENIED`,没有把无权读取改写成状态文件或本地批次不存在。该权限边界使生产侧原始失败 phase 保持未知,不据此猜测具体 stderr。
ARM 侧结果:dead-man timer 与公网健康 timer 均为 `active`;dead-man service 最近一次按失败状态退出。实际 dead-man 阈值为显式参数 `93600` 秒(26 小时)。`last-deadman` 为 `FAIL_OFFHOST_DEADMAN/archive_stale/findingCount=1`,alert state 同为 `FAIL_OFFHOST_DEADMAN/archive_stale`。最近 72 小时 journal 仅做 allowlist 聚合,得到 `sent=true` 一次、`sent=false` 两次,最近一次为 `sent=true`。该次 dead-man timer 触发锚点为 2026-08-27 06:30:06(Asia/Shanghai),alert state 在同轮随即更新,可作为本次发送调用时刻。恢复演练探针在同日稍后首先促成人工响应,但当时未独立采集“人员实际知晓”的精确墙钟时刻,因此本轮不能伪造精确响应延迟,只能登记为“发送调用早于人工响应、精确差值未测”。后续事件任务必须同时记录 `sent=true` 与首次确认时刻,直接计算真实响应延迟。
检测侧结论为正面:真实超龄被正确检测,告警 helper 至少完成了一次发送调用,且相同失败的 `sent=false` 属于正常去抖;08-23 去抖改造没有造成检测或发送调用盲区。响应侧仍有 finding:本次备份中断最终由恢复演练探针触发处置,而不是由既有告警形成及时响应。证据不证明具体人员已阅读消息,后续应调查群消息送达、可见性、责任人确认和升级流程,不能把检测调用成功等同于闭环响应。
receiver 聚合结果:正式原子日归档共 7 个,最新归档在复核时年龄约 43 小时 40 分;inbox 有 2 个 `.partial`,正式归档事务临时对象为 0、inbox 其他对象为 0。两个 partial 均不满足“三个普通文件 + manifest/run 身份 + 产物字节数”的完整传输合同,完整合同数量为 0、不完整数量为 2;最新 partial 年龄约 19 小时 40 分,最旧 partial 约 31 天 10 小时。由此将当前事故分类为 `offhost_transfer_incomplete`:生产 daily 确实触发并在 ARM 留下未完成传输,不能归为源端完全未产出、timer 漏调度或“传输已经完整但 finalize 失败”。partial 的存在同时证明 rsync 已通过初始 SSH 认证、host key 校验和 receiver 授权并开始写入,因此密钥失效、authorized_keys 初始拒绝和 host key 初始拒绝不是本次首要候选;排查应收缩到传输中段的超时/网络中断、源端读取、目标端写入或会话/传输限制。最旧 partial 同时是历史清理/观测 finding,但本轮没有删除它。
当前只读字段还不足以在上述传输中段候选之间继续裁决。下一轮独立生产事件调查应比较两个 partial 的聚合字节大小/停止比例,读取 wrapper 私有诊断中的真实 rsync 退出码,并记录每轮传输持续时间:相近停止偏移或相近时长支持确定性限制,不同偏移/时长更支持网络或超时;service 的通用 exit `1` 不得替代 rsync 原始码。该调查仍只输出字节聚合、退出码和耗时,不输出文件名、路径或 stderr 原文。
容量必须与传输修复并行处理:生产侧相关路径位于同一可识别文件系统,使用率约 82%、可用约 10 GiB;ARM archive/inbox 文件系统使用率约 53%、可用约 89 GiB。现有证据仍不足以把这次 rsync 中断直接归因于磁盘不足,但自动备份会先在生产生成完整 dump 与 storage 归档,持续失败还会反复生成/保留本地产物;10 GiB 对该流程属于薄余量。若继续下降,故障可能从“传输中断”升级为“源端无法产出”,因此容量治理不是低优先级后续项,而是独立生产事件中的并行处置项。
调查器本身出现并安全关闭了两类本地缺陷:首次生产探针错误假设部署账号可 sudo,安装前即被权限拒绝;前两次 ARM 探针把 dead-man unit 写成近似但不存在的名称,均在状态读取前以合同不可读失败。每次 root 入口、sudoers candidate、普通用户暂存均为 0。最终版本从仓库 receiver 安装器机械提取并比较 unit basename,并将生产探针改为无提权只读执行;Bash/Python 语法、两种 ExecStart 解析 fixture 和敏感扫描通过。上述缺陷不改变生产、ARM、归档、timer 或告警状态,也没有伪造调查结果。
当前独立生产 finding 为:最新可用异机恢复点已超过 24 小时目标,自动备份传输链正在失败;恢复演练因此保持阻塞。该事件不再作为恢复演练的普通 finding 排队处理,建议另建 Trellis 生产事件任务承载传输/容量调查与修复,当前演练任务仅保留证据交叉引用和阻塞关系。后续优先顺序是另行批准修复生产到 ARM 的传输链并形成新的原子批次,确认自动备份/两端 dead-man 与 watchdog 恢复,再对新批次重新执行身份探针。手工补跑备份、修改网络/SSH/timer、清理 partial 或发送测试告警均不在本次只读批准内。