完善异机备份告警与同日恢复点取代（07-27-automatic-offhost-backup）

2026-07-27 13:54:16 +08:00
@@ -927,13 +927,18 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
 - Production runner: `scripts/run-automatic-backup.sh` with `run`,
  `check-stale` and `remind-drill` modes.
 - ARM forced receiver: `scripts/receive-offhost-backup.sh`.
+- Shared host alert helper: `scripts/send-wecom-group-alert.py` with explicit
+  `--webhook-file`, `--summary-file` and `--state-file` paths.
+- ARM watchdog: `scripts/check-offhost-backup-deadman.py` with archive root,
+  state root, alert helper, Webhook file and a default `93600` second threshold.
 - Installers: `install-automatic-backup-timer.sh` and
  `install-offhost-backup-receiver.sh`.
 - Local contract: `bash scripts/qa/run-automatic-backup-contract.sh`.
 - Production required environment: `REMOTE_DIR`, `REMOTE_ENV_FILE`,
  `COMPOSE_PROJECT_NAME`, `POSTGRES_DATA_DIR`, `STORAGE_HOST_DIR`, `BUILD_SHA`,
  `VITE_BUILD_SHA`, `AUTOMATIC_BACKUP_DIR`, `AUTOMATIC_BACKUP_STATE_DIR`,
-  `OFFHOST_TARGET`, `OFFHOST_SSH_KEY` and `OFFHOST_KNOWN_HOSTS`.
+  `OFFHOST_TARGET`, `OFFHOST_SSH_KEY`, `OFFHOST_KNOWN_HOSTS`,
+  `ALERT_WEBHOOK_FILE` and `ALERT_HELPER`.
 - Receiver config: independent absolute `BACKUP_INBOX_DIR`,
  `BACKUP_ARCHIVE_DIR`, `BACKUP_WEEKLY_DIR`, `RRSYNC_BIN`, `PYTHON_BIN`,
  `DAILY_KEEP` and `WEEKLY_KEEP`.
@@ -952,14 +957,32 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
  it rejects shell, sender/read and destructive rsync options.
 - ARM revalidates manifest identity, exact regular-file set, sizes and SHA-256,
  then atomically moves the batch to archive. Archive and weekly directories are
-  outside the rsync root; an archived name is never overwritten.
+  outside the rsync root. A different run ID for the same UTC day may replace
+  that day's archive only after full candidate validation: rename the old daily
+  point to a same-filesystem rollback name, publish the candidate, rebuild that
+  ISO week's hard-link checkpoint, then remove the rollback. Any publication or
+  weekly-checkpoint failure restores the prior daily point.
 - Write production `last-success.json` and prune source copies only after ARM
  finalize succeeds. Keep seven automatic source/ARM daily points and four ARM
  weekly checkpoints; unrelated manual batches are outside automatic retention.
 - Daily, freshness and quarterly reminder timers are separate. Alert summaries
-  contain categories only. AUTO_BACKUP sends the first failure, worsening/new
-  findings and the first quarterly reminder, while suppressing an identical
-  repeated failure or reminder.
+  contain categories only. A shared Python helper reads the complete WeCom group
+  robot URL only from a regular, executing-account-owned `0600` file and never
+  prints it. It sends the first failure, worsening/new findings, one recovery
+  notice and the first quarterly reminder, while suppressing an identical
+  repeated state. It does not call the OA server container or resolve an employee.
+- ARM runs its own 26-hour dead-man as root. It verifies the newest archive's
+  exact file set, manifest identity, sizes, checksums and timestamp, so a stopped
+  production host/server can still cause a group alert. Missing, corrupt, stale
+  or future-dated archives fail; recovery sends once.
+- Production command diagnostics for paired backup, rsync and finalize stay only
+  in the state directory as `last-error.log` mode `0600`. The summary and group
+  message contain only the safe phase category. A later successful off-host run
+  removes the retained diagnostic.
+- The production freshness check hard-fails missing, malformed, future, stale or
+  Compose-project-mismatched state. A fresh recovery point with a different
+  build SHA is `PASS_WITH_FINDINGS`, because deploy-time code identity and
+  recoverable data freshness are separate facts.
 - Installer `install` is disabled-by-default; `enable`, real transfer, real
  WeCom alert and all server/account/key changes remain `EXTERNAL_GATED`.

@@ -967,24 +990,32 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \

 - Source mount/build/project drift, lock conflict or incomplete backup -> fail
  before success state and source retention.
- Transfer/finalize failure, wrong/extra/link artifact, bad hash or second batch
-  for one UTC day -> leave no off-host success state and preserve source batch.
- Missing, stale or project/build-mismatched success record -> watchdog finding.
+- Transfer/finalize failure, wrong/extra/link artifact or bad hash -> leave no
+  off-host success state, preserve the source batch and retain a root-only local
+  diagnostic. A corrupt same-day candidate leaves the previous archive intact.
+- Valid same-day retry -> replace only that UTC day's point and rebuild its ISO
+  weekly checkpoint; concurrent finalize -> `flock` rejection.
+- Missing, corrupt, stale, future or project-mismatched success record -> hard
+  watchdog failure. Fresh build mismatch -> one finding without a failing exit.
+- Missing/non-regular/symlink/non-`0600` Webhook file -> fail closed without
+  printing the file value. Identical alert state -> update/evaluate without send.
 - Ordinary shell, rsync sender/read/delete or archive overwrite -> receiver
  rejection without changing archive.
- Host/network/WeCom all unavailable at once -> self-hosted alert cannot prove
-  its own outage; this is a documented physical boundary, not a green state.
+- ARM host/network/WeCom all unavailable at once -> the group robot path cannot
+  prove its own outage; this physical boundary is not a green state.

 ### 5. Good/Base/Bad Cases

 - Good: one verified active source creates a complete batch, write-only rsync
  resumes inside inbox, ARM finalize verifies and archives it, then production
-  writes success and prunes old automatic source points.
+  writes success and prunes old automatic source points. If production later
+  stops, ARM independently raises one dead-man alert and one recovery notice.
 - Base: local implementation and synthetic contracts pass while both server
  installers remain disabled and explicitly externally gated.
- Bad: give production a normal ARM login key, put archive below inbox, accept
-  a new host fingerprint automatically, or prune local recovery points after
-  rsync but before target-side finalize.
+- Bad: send through `compose exec server`, hard-code a personal employee number,
+  persist a robot URL in `.env`/unit/repository, give production a normal ARM
+  login key, put archive below inbox, accept a new host fingerprint automatically,
+  or prune local points before target-side finalize.

 ### 6. Tests Required

@@ -993,28 +1024,37 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
 - Docker mocks must return the real `{{json .Mounts}}` shape consumed by the
  runner. A simplified tab-delimited mock is not valid evidence for real mount
  parsing.
- Focused WeCom alert tests for stdin path redaction, preserved AUTO_BACKUP
-  issue identity, first/repeated failure and quarterly reminder behavior.
- Environment contract assertions for all four scripts, three timers, explicit
-  project/data/build/SSH binding and disabled-by-default installation.
- Real ARM setup, first paired batch, failure alert and quarterly shadow restore
-  are separately approved operational evidence.
+- Contract assertions for same-day replacement, corrupt-candidate preservation,
+  weekly hard-link rebuild, source diagnostic mode, build mismatch finding,
+  Webhook mode/symlink rejection and group alert behavior without a server
+  container. Test transport must be explicitly gated and local-only.
+- Environment contract assertions for both helpers, both installers, production
+  timers, ARM dead-man timer, explicit project/data/build/SSH/Webhook-file
+  binding, disabled-by-default installation and absence of a literal complete
+  robot URL in tracked or untracked repository files.
+- Real ARM setup, first paired batch, deliberate source-timer stop or transfer
+  failure, dead-man first/repeated/recovery behavior and quarterly shadow restore
+  are separately approved operational evidence. A direct robot send test alone
+  is not dead-man evidence.

 ### 7. Wrong vs Correct

 #### Wrong

 ```bash
+compose exec server send-alert --recipient=<personal-employee-number>
 rsync -a <batch>/ backup-host:<archive>/
-rm -rf <old-local-batches>
 ```

 #### Correct

 ```bash
-# Install only with explicit active identity; default remains disabled.
-<explicit-project-data-build-and-ssh-env> \
+# Install only with explicit active identity and a root-only Webhook file path;
+# default remains disabled.
+<explicit-project-data-build-ssh-and-webhook-file-env> \
  bash deploy/trialrun/install-automatic-backup-timer.sh install
+# ARM separately verifies archive freshness even when production is unavailable.
+python3 scripts/check-offhost-backup-deadman.py <explicit-safe-path-arguments>
 # The runner writes success and prunes only after restricted target finalize.
 bash scripts/run-automatic-backup.sh run
 ```
@@ -19,7 +19,12 @@ production 02:30
  -> prune local daily copies

 production 07:10 watchdog
-  -> last off-host success <= 24h ? pass : alert
+  -> recovery point missing/corrupt/stale/project drift ? fail + group alert
+  -> build SHA differs ? one finding + group alert
+
+ARM independent watchdog
+  -> newest finalized archive <= 26h ? pass : direct group alert
+  -> source host/server/container not required

 quarterly reminder
  -> one redacted WeCom reminder per calendar quarter
@@ -39,8 +44,8 @@ quarterly reminder
 - ARM 一次性安装 rsync，创建密码锁定的专用账号、独立 inbox/archive/weekly 目录和一把专用公钥。该公钥的 `authorized_keys` 强制执行仓库提供的 receiver，并禁用 PTY、端口转发、agent 转发和 X11 转发。
 - receiver 只接受两类 `SSH_ORIGINAL_COMMAND`：rsync server 接收模式，或 `hyb-backup-finalize <run-id> <manifest-sha256>`。其他命令全部拒绝。
 - rsync 通过 `rrsync -wo` 只能进入 inbox；archive 和 weekly 位于 rrsync 根之外。即使生产端密钥泄露，也不能用 rsync 读取或删除已归档批次。
- finalize 只接受 `auto-` 安全批次号和完整 manifest SHA。它要求 inbox 中恰有三个普通文件、无链接，manifest 的 runId/status/schema、相对文件名、大小和 SHA-256 全部匹配；已存在 archive 目标时拒绝覆盖。
- 校验通过后在同一文件系统内原子移动到 archive。每个自然日最多接受一个正式自动批次进入保留计算，避免失陷源端通过短时间伪造大量批次提前挤掉历史。
+- finalize 只接受 `auto-` 安全批次号和完整 manifest SHA。它要求 inbox 中恰有三个普通文件、无链接，manifest 的 runId/status/schema、相对文件名、大小和 SHA-256 全部匹配；同 run_id 重复 finalize 继续保持幂等。
+- 同 UTC 日已有不同 run_id 时，先完整校验并冻结新批次权限，再把旧日归档 rename 到同文件系统临时回滚名、新批次 rename 到正式归档名；新批次发布失败立即把旧归档 rename 回原名。新批次发布成功后才删除旧回滚名，并重建该 ISO 周硬链接检查点，使每日一个恢复点和周检查点都指向当前有效批次。
 - archive 保留最近 7 个日批次；每个 ISO 周首次成功批次用只读硬链接快照形成周检查点，保留最近 4 周。清理不得删除最后一个 archive 或当前 finalize 批次。
 - 旧手工/审计批次不进入自动前缀和自动轮转范围。

@@ -52,19 +57,22 @@ quarterly reminder

 ## 告警与静默失败检测

- wrapper 无论成功或失败都生成字段白名单 summary；失败 trap 尝试通过活动 server 容器调用现有企微运维告警 CLI，并使用备份专属 state file，避免与 WARCH3 降噪状态互相覆盖。
- 现有告警 CLI 小范围泛化为接受 stdin summary、保留 summary issueId，并为备份失败/过期提供正确建议；发送仍受既有 `WARCH3_OPS_ALERT_SEND=1` 双门控制。
- 独立 watchdog timer 在每日巡检前读取宿主 `last-success.json`。文件缺失、格式错误、构建/项目不匹配或成功时间超过 24 小时均产生失败 summary 并尝试告警。
- 若 Docker、整机网络或企微本身同时不可用，主机无法自证并外发，这是自托管告警的物理边界；watchdog 能覆盖 timer 未运行和短暂失败，但不替代外部主机存活监控。
+- 新增一个小型宿主告警 helper，使用 Python 标准库从 root-only Webhook 文件读取完整 URL，并向企微运维群机器人发送固定白名单文本；不复用 OA server 容器、员工表或个人工号。
+- 生产 wrapper 无论成功或失败都生成字段白名单 summary；即时失败和源端 watchdog 直接调用宿主 helper。Webhook 文件缺失、权限不是 `0600`、不是普通文件或内容格式不合法时 fail closed，并在本机诊断中留类别，不打印 URL。
+- ARM receiver 安装器同时安装独立 dead-man service/timer。watchdog 读取 archive 中最新有效自动批次的完成时间，超过 26 小时、没有批次或 manifest 无法验证时直接调用同一 helper；状态文件实现首次失败、重复降噪和恢复通知。
+- 生产 stale watchdog 仍验证 schema/status/project/时间。build SHA 不同输出 `PASS_WITH_FINDINGS`，不返回失败；恢复点缺失、损坏、未来时间、超过阈值或 project 不同继续返回失败。
+- 生产外部命令的 stdout/stderr 只写状态目录下的临时诊断，失败时原子保留为 `last-error.log` 并设为 `0600`，成功后清理本次临时诊断。summary 和企微文本不得复制诊断正文。
+- Webhook secret 同时存在于两台主机的 root-only 文件，仓库只保存“文件路径”合同；`.env.example` 和 systemd `Environment=` 都不得出现 URL/key。
 - 季度提醒以年份+季度作为 finding key，确保每季度一次且同季度不重复；只提醒人工影子恢复，不执行恢复。

 ## 安装与回滚

 1. 本地先实现合成测试、受限 receiver、告警适配和两个安装器，不接触生产/ARM。
-2. 获批后先在 ARM 安装 rsync、受限账号/receiver/公钥和目标目录；使用无业务数据的合成批次验证拒绝 shell、拒绝 archive 覆盖和 finalize。
-3. 再在生产安装专用私钥/known_hosts、每日备份 service/timer、stale watchdog 和季度提醒；首次安装先保持 timer disabled。
-4. 手工启动一次获批自动备份，核对源/ARM 哈希、状态和无敏感输出；再 enable timers。
-5. 用不包含业务正文的可控失败验证企微告警后恢复链路，最终确认下次执行时间。
+2. 获批后先由用户在企微创建运维群机器人，并分别把同一 Webhook 写入两台主机 root-only 文件；不向 Codex、仓库或普通 `.env` 提供其内容。
+3. 在 ARM 安装 rsync、受限账号/receiver/公钥、dead-man timer 和目标目录；使用无业务数据的合成批次验证拒绝 shell、同日安全取代和 dead-man 状态逻辑。
+4. 再在生产安装专用私钥/known_hosts、每日备份 service/timer、stale watchdog 和季度提醒；首次安装先保持 timer disabled。
+5. 手工启动一次获批自动备份，核对源/ARM 哈希、状态和无敏感输出；再 enable timers。
+6. 真正停止一次源端 timer 或故意让一次传输失败，等待/模拟超过 26 小时并确认 ARM 群告警；恢复链路后验证恢复通知和新批次成功。

 回滚只 disable/stop 新 timer、移除 unit 和专用密钥授权；保留已生成的完整备份、状态报告和 ARM archive，不删除历史数据。旧手工 `backup-all.sh` 入口保持兼容。

@@ -74,3 +82,4 @@ quarterly reminder
 - 不给生产普通 ARM shell，而用 inbox + 强制 receiver + archive 隔离，抵消推送方案最主要的跨机删除风险。
 - 使用 rsync 而不是新备份服务；当前压缩批次规模较小，收益主要是断点续传和成熟 SSH 传输，不追求块级去重。
 - 不自动恢复。季度人工影子恢复才是可恢复性证据，日 timer 只负责生成恢复点。
+- 运维加固在 dead-man、真实失败链路、timer、恢复卡第二人走查完成后硬收口；不继续新增平台化备份或告警能力，后续回到产品 backlog。
@@ -9,9 +9,16 @@
 - [x] 增加合成契约测试：来源漂移、锁冲突、不完整/损坏/重复批次、续传 partial、拒绝普通 shell、保留轮转、stale 和季度降噪。
 - [x] 更新部署环境契约、runbook、checklist、管理员手册的最小操作入口和 `.trellis/spec/hyb-oa/deployment-trialrun.md`；不新增平行长文档。
 - [x] 生成一页脱敏恢复卡内容，优先放在现有 runbook 的首部紧急段；由用户复制到公司知识库并安排第二人走查。
- [ ] 完成本地全量门禁并提交；生产、ARM 安装另行请求批准。
+- [x] 完成本地全量门禁并提交；生产、ARM 安装另行请求批准。
 - [ ] 获批后先安装 ARM 接收端并用合成批次验证，再安装生产 units，手工运行首次真实自动批次，最后 enable timers。
 - [ ] 获批执行一次可控传输失败告警并恢复正常链路；不读取附件正文、不执行业务或企微其他写入。
+- [x] 删除个人工号默认值，新增生产/ARM 共用的 root-only 企微群机器人 helper；两端 systemd 只绑定 Webhook 文件路径。
+- [x] 增加 ARM 26 小时 dead-man service/timer，覆盖缺批次、超龄、损坏、重复降噪和恢复通知。
+- [x] 将 buildSha 不同降级为 finding；缺失/损坏/超龄/project 不同继续硬失败，并支持部署后立即运行一次备份。
+- [x] 为 backup/rsync/finalize 保存 `0600` 最近失败诊断，summary 与群消息只保留脱敏类别。
+- [x] 把同 UTC 日不同 run_id 从拒绝改为“完整校验后原子取代”，覆盖失败回滚和周检查点重建。
+- [x] 增加 Webhook 不进入 `.env.example`/仓库配置/报告的否定断言，并在 runbook 记录调整 `BACKUP_RUN_AT` 后需人工确认次日窗口。
+- [x] 本地门禁通过并完成只读复核后提交；不推送、不部署、不配置真实 Webhook。

 ## 预计文件边界

@@ -22,6 +29,7 @@
 - 新增 `scripts/qa/run-automatic-backup-contract.sh`：纯合成正负向测试。
 - 修改 `server/src/scripts/send-wecom-ops-alert.js` 及其单测，仅做通用脱敏 summary 兼容。
 - 修改 `deploy/trialrun/validate-env-contract.js`、现有 runbook/checklist/管理员手册和部署规约。
+- 新增一个不含 Secret 的宿主告警 helper；只读取安装时传入的 root-only Webhook 文件。

 ## 验证命令

@@ -37,28 +45,31 @@
 - `node scripts/qa/audit-docs.mjs --check-registry`
 - `node scripts/qa/check-process-contract.mjs --baseline --json`
 - `git diff --check`
+- 合成测试必须模拟源端不再产生 finalize，并由 ARM dead-man 得到失败；不能只直接调用“发送消息”路径。

 ## 外部门禁与回滚点

 - 安装 ARM rsync/账号/authorized_keys/目录属于远端配置写入，须单独批准；不得复用普通登录账号或本地工作站私钥。
 - 安装生产 SSH key/config/systemd units、首次真实备份、真实异机传输和企微失败告警均须单独批准。
+- 创建企微运维群机器人及把 Webhook 写入两台主机 root-only 文件由用户手工完成；Codex 不读取、不回显、不记录其内容。
 - 首次真实执行前记录现有手工批次数量、磁盘余量、活动容器/mount/build 和现有 timer；只输出脱敏计数/布尔值。
 - 任何目标身份、主机指纹、活动 mount、manifest 或哈希不匹配立即停止，不猜路径、不接受新指纹、不清理源端批次。
 - 回滚新 timer/receiver 时只撤销调度和专用授权；ARM archive、源端完整批次和原 WARCH3 timer 保留。

 ## 本地实施与验证证据

- 已实现生产 wrapper、ARM forced receiver、两端安装器和 3 组 systemd timer；安装器默认只安装 disabled units。
- `run-automatic-backup-contract.sh` 使用临时目录与 mock Docker/SSH/rsync 完成 28 项正负向合同，不连接真实主机。
+- 已实现生产 wrapper、ARM forced receiver、独立群机器人 helper、ARM dead-man 和两端安装器；生产 3 组 timer 与 ARM 1 组 timer 均默认只安装为 disabled。
+- `run-automatic-backup-contract.sh` 使用临时目录与 mock Docker/SSH/rsync 完成 46 项正负向合同，不连接真实主机；覆盖同日取代与回滚、坏候选保留、周检查点重建、Webhook 权限/链接/格式负例、build finding、状态损坏、源端停止、重复降噪和恢复通知。
 - 企微告警聚焦单测 15 项通过；stdin 路径被替换为安全类别，AUTO_BACKUP 首次失败/季度提醒发送且相同重复状态降噪。
 - `run-backup-restore-drill.sh` 本地合成 DB/附件往返通过，本次合成 RTO 为 8 秒；该结果不是生产 RTO 或 ARM 恢复证据。
 - `scripts/check-minimal.sh` 通过：server 162 个测试文件、1004 项测试全部通过，client typecheck/build 通过。
 - 环境合同、Trellis task/governance、敏感扫描、文档注册、流程合同和 `git diff --check` 均通过；文档注册保留既有 22 个未注册文件 warning，无本任务新增文档。
- ARM 安装、生产 units、真实首批、timer enable、真实企微失败告警和第二人季度走查仍为外部门禁，未执行、不得写成已完成。
+- 最终只读复核修正了三个本地缺陷：生产安装器残留个人工号校验、ARM 重装可能改变 dead-man 状态文件属主、stale 状态未校验 run id/manifest SHA；修正后 46 项合同与 `scripts/check-minimal.sh` 全量复跑通过。
+- 更新后的 ARM dead-man/helper、生产 helper/runner、真实有效首批、四组 timer enable、真实企微失败链路和第二人季度走查仍为外部门禁，未执行、不得写成已完成。

 ### 首次远端执行增量

 - ARM 已安装 rsync、密码锁定专用账号和 forced-command receiver；从生产验证普通 shell、sender/read、delete 均拒绝，写通道 dry-run 通过且未生成归档。
 - 生产三组 units 已安装为 disabled/inactive，并绑定活动 Compose project、PGDATA、附件挂载、构建标识和固定 SSH 身份。
- 首次真实备份在 `runtime_identity` 阶段 fail-closed，未生成源端或 ARM 批次；真实 Docker `println` 输出的参数间空格导致制表符拆分后路径不精确相等。
- runner 已改为解析 `docker inspect` mount JSON 并要求唯一匹配；合成 mock 同步改为真实 JSON 形状。重新执行真实首批和 timer enable 尚待验证。
+- 首次真实备份曾在 `runtime_identity` 阶段因 Docker `println` 参数间空格 fail-closed；runner 改用 `docker inspect` mount JSON 后，后续真实源端成对批次已生成并传输。
+- ARM finalize 随后因同 UTC 日已有不同 run id 而拒绝，因此尚无可作为自动备份闭环证据的新批次；本轮已在本地实现“完整校验后同日取代”，更新后的远端脚本、首批和 timer enable 仍待另行批准。
@@ -12,6 +12,8 @@
 - 两台主机之间当前没有可直接复用的免密 SSH 别名或密钥链路，不能把本地工作站的 SSH 配置误当作服务器间自动化能力。
 - 当前手工正确批次和 ARM 影子恢复已证明格式可用，但不能满足 RPO 24 小时或自动失败发现。
 - 已确认采用“生产主动推送到 ARM 的专用受限接收身份”：生产保留即时传输结果，ARM 的已归档批次不暴露给 rsync 写入根。
+- 生产切换已经完成并应用 61 项 migration；本地当前为 62 项，`2526052` 后续部署只新增 1 项 migration。自动备份三组 units 已安装但仍未形成可靠启用闭环。
+- 首次真实自动批次已证明源端成对备份可生成，但 ARM 因“同一 UTC 日仅允许一个不同 run_id”拒绝重试批次；现有即时告警还依赖活动 server 容器，不能覆盖容器或整机故障。

 ## Scope

@@ -28,7 +30,11 @@
 - 完整批次通过固定主机指纹和专用受限密钥自动推送到 ARM；该身份无普通 shell，只能把文件写入 inbox 或请求校验/归档，不能通过 rsync 读取或删除 archive。
 - 传输端和目标端都要校验 manifest、文件大小和 SHA-256；同名已归档目标不得覆盖。中断传输可以在 inbox 内续传，但只有完整校验后才能原子进入 archive。
 - 生产保留最近 7 个成功日批次；ARM 至少保留最近 7 个日批次和 4 个周检查点。清理只能发生在本次异机成功确认之后，且不得删除最后一个已知成功批次。
- 备份或异机校验失败必须生成脱敏状态报告，并复用现有企微运维告警能力；独立 stale watchdog 应在现有 07:30 巡检前检查最近一次异机成功时间，防止备份 timer 未运行或即时告警自身失败造成静默缺口。
+- 生产即时告警和 ARM dead-man 告警统一使用独立企微运维群机器人；Webhook 只从 root-only 文件读取，不依赖 OA server、数据库、员工绑定或默认个人工号，也不得进入仓库、`.env.example`、Trellis、日志或聊天。
+- ARM 安装独立 26 小时 dead-man timer，直接检查最近归档完成时间；生产整机或容器不可用、timer 未运行、传输未 finalize 时仍能从另一台主机发出告警。
+- 生产 watchdog 的“恢复点缺失/损坏/超龄/Compose project 不匹配”继续硬失败；仅 `buildSha` 早于当前部署时降级为一次 finding，不得误报为没有恢复点。部署后仍推荐立即触发一次获批备份。
+- backup、rsync 和 finalize 的失败输出必须写入状态目录下权限 `0600` 的最近失败诊断；企微告警仍只包含阶段、状态、finding 数和脱敏建议，不附原始输出。
+- 同一 UTC 日已有不同自动归档时，新批次必须先完整通过 manifest、文件集、大小和 SHA-256 校验，再以可回滚的同文件系统 rename 顺序原子取代旧日批次；任一替换步骤失败必须恢复旧批次或保留可诊断状态。最终仍只保留每日一个恢复点。
 - 告警只包含阶段、状态、finding 数和脱敏建议，不包含数据库凭证、完整目录、SSH 信息、真实业务数据、附件名或原始命令输出。
 - 季度恢复保持人工触发，在 ARM 隔离目标执行；季度 timer 只发送一次脱敏提醒，不得自动运行恢复，更不得触碰生产库。
 - 形成一页脱敏恢复卡，说明主机故障后的负责人、备份位置类别、恢复顺序和验收项，供另一名人员存入公司知识库。
@@ -38,7 +44,11 @@
 - [x] 本地合成测试证明每日 wrapper 只接受唯一活动来源、互斥锁有效，并只发布完整备份批次。
 - [x] 传输测试证明不完整批次、哈希不一致、重复批次和不可达 ARM 均失败，且不产生异机成功状态。
 - [x] 保留测试覆盖 7 个日批次、4 个周检查点、最后成功批次保护和仅在异机确认后清理。
- [x] 告警测试证明首次失败和状态恶化会告警，重复同一失败降噪；07:30 stale 检查能发现超过 24 小时没有异机成功记录。
+- [x] 告警测试证明首次失败和状态恶化会告警，重复同一失败降噪；07:10 stale 检查能发现超过 24 小时没有异机成功记录。
+- [x] 本地合成测试证明同日不同 run_id 的完整新批次可取代旧批次，损坏新批次不会影响旧归档，且周检查点与日归档不产生悬空或错误指向。
+- [x] 本地合成测试证明 buildSha 不同仅产生 finding，恢复点缺失/损坏/超龄/project 不同仍失败；失败诊断为 `0600` 且不进入告警摘要。
+- [x] 环境契约证明 Webhook 相关键和值不出现在 `.env.example`、仓库配置或报告；两端安装器只接受 root-only Webhook 文件路径。
+- [x] ARM 26 小时 dead-man 合成验证覆盖“源端停止/不产生新 finalize”而非仅验证机器人能发送，并覆盖首次失败、重复降噪与恢复通知。
 - [ ] systemd 安装器显式绑定 Compose project、活动数据路径、构建标识、备份根和受限远端身份；安装后 timer enabled/active，但部署验证不手工执行真实告警。
 - [ ] 经单独生产授权后完成一次自动日备份：源端完整、ARM 目标端哈希一致、状态报告脱敏，且未读取附件正文或修改业务数据。
 - [ ] 模拟传输失败能在当天产生一条脱敏企微告警；测试后恢复正常链路并完成成功批次。
@@ -49,12 +59,15 @@
 - 不引入备份平台、对象存储、MinIO、云厂商 SDK、Kubernetes 或持续复制。
 - 不自动执行生产恢复、数据库 reset、迁移改写或附件回写。
 - 不把真实业务正文、附件清单、凭证、主机地址、完整目录或 SSH 私钥写入仓库、报告或聊天。
+- 不把企微群机器人 Webhook URL、key 或配置文件正文写入仓库、`.env.example`、Trellis、报告、日志或聊天。
 - 不在本任务继续企微本人授权、员工角色、付款、报销或其他产品功能。
 - 不因自动备份完成而取消季度人工影子恢复。

 ## Test Plan

 - Shell 语法与合成文件系统测试，覆盖锁、来源解析、传输失败、目标重复、哈希和保留轮转。
+- 合成 receiver 测试覆盖同日完整批次取代、损坏候选保留旧批次、替换失败恢复以及周检查点一致性。
+- 合成双主机告警测试覆盖生产无 server 容器、ARM 26 小时无新归档、重复降噪、恢复通知和 root-only Webhook 文件门禁。
 - 复用 `scripts/qa/run-backup-restore-drill.sh` 验证成对备份合同未回退。
 - 聚焦测试现有企微运维告警的备份状态与 stale finding 适配。
 - `node deploy/trialrun/validate-env-contract.js`。
@@ -31,7 +31,7 @@
 - [ ] 正式 deploy 使用绝对数据路径；既有实例使用 `DEPLOY_DATA_MODE=existing`，PGDATA 已存在且含 `PG_VERSION`，附件目录已存在
 - [ ] 已证明没有两个 PostgreSQL 实例同时挂载同一 PGDATA；任何不一致均停止发布
 - [ ] 已记录活动 server/db 容器、不可变镜像 ID（及可用 repo digest）、Caddy 目标、挂载源、环境摘要，不记录 env 正文
- [ ] 已冻结成功 migration 数量（当前预期 58）和关键业务表逐表计数基线
+- [ ] 已现场只读冻结成功 migration 数量和关键业务表逐表计数基线；当前生产证据为 61 项、本地为 62 项，本次待上线只应新增 1 项，现场结果不符立即停止
 - [ ] 确认当前版本已打 tag：`git tag -a trialrun-YYYYMMDD-NN -m "trialrun release"`
 - [ ] 本地门禁：`./scripts/check-minimal.sh`
 - [ ] 部署契约校验：`node deploy/trialrun/validate-env-contract.js`
@@ -48,12 +48,14 @@
  - [ ] 完整批次已复制到获批异机/异盘，并记录保留周期
  - [ ] 同批次已在隔离影子目标恢复：migration 数、关键业务表逐表计数、附件普通文件数均与冻结基线一致，兼容镜像 `/ready` 通过
 - [ ] 自动异机备份（未完成远端门禁前保持未勾选）：
-  - [ ] **EXTERNAL_GATED**：ARM 已获批安装 rsync、密码锁定专用账号、forced-command key 和独立 inbox/archive/weekly；普通 shell、rsync sender/read/delete 均被拒绝
-  - [ ] **EXTERNAL_GATED**：生产专用私钥为 `0600`，known-hosts 指纹已人工核对，安装器显式绑定活动 project、PGDATA、附件、完整/短构建 SHA、备份/状态目录和远端身份
+  - [ ] **EXTERNAL_GATED**：用户已在独立企微运维群创建机器人，并分别把 Webhook 写入生产、ARM 的 root-only `0600` 文件；未向 Codex、仓库、普通 `.env`、日志或聊天提供正文
+  - [ ] **EXTERNAL_GATED**：ARM 已获批安装 rsync、密码锁定专用账号、forced-command key、独立 inbox/archive/weekly 和 26 小时 dead-man；普通 shell、rsync sender/read/delete 均被拒绝
+  - [ ] **EXTERNAL_GATED**：生产专用私钥为 `0600`，known-hosts 指纹已人工核对，安装器显式绑定活动 project、PGDATA、附件、完整/短构建 SHA、备份/状态目录、远端身份和 Webhook 文件路径
  - [ ] `install-automatic-backup-timer.sh install` 只安装 disabled units；首批真实批次及 ARM manifest/SHA-256 核对通过后才显式执行 `enable`
-  - [ ] daily/watchdog/drill-reminder 三组 timer enabled/active，daily service 当前 inactive；未通过手工启动真实告警来伪造验证
+  - [ ] 生产 daily/watchdog/drill-reminder 与 ARM dead-man 四组 timer enabled/active，各 oneshot service 当前 inactive；未通过只调用机器人发送入口来伪造验证
  - [ ] 最近一次异机成功不超过 24 小时；生产自动日批次为 7 个，ARM 自动日批次为 7 个、周检查点为 4 个，历史手工批次未被轮转
-  - [ ] **EXTERNAL_GATED**：可控传输失败仅产生一条脱敏企微告警，恢复链路后新批次成功；同一失败重复运行已降噪
+  - [ ] **EXTERNAL_GATED**：已真正停止一次源端 timer 或故意破坏一次传输，ARM dead-man 仅产生一条脱敏企微告警；恢复 finalize 后收到一次恢复通知，同一失败重复运行已降噪
+  - [ ] 若调整过 `BACKUP_RUN_AT`，已人工确认调整后的次日 UTC 分桶和 26 小时窗口，未把一次预期空窗误判为长期故障
  - [ ] 本季度已由第二名内部人员按 runbook 首部恢复卡在 ARM 隔离目标完成影子恢复走查；timer 未自动恢复生产
 - [ ] 停机复制方案已冻结：旧 PostgreSQL 完全停止后复制到空新目录并保留所有权/权限/链接/扩展属性；旧目录原位保留且观察期内不再写入

@@ -14,7 +14,8 @@ AUTOMATIC_BACKUP_STATE_DIR="${AUTOMATIC_BACKUP_STATE_DIR:-}"
 OFFHOST_TARGET="${OFFHOST_TARGET:-}"
 OFFHOST_SSH_KEY="${OFFHOST_SSH_KEY:-}"
 OFFHOST_KNOWN_HOSTS="${OFFHOST_KNOWN_HOSTS:-}"
-ALERT_RECIPIENT_EMPLOYEE_NO="${ALERT_RECIPIENT_EMPLOYEE_NO:-HYD000006}"
+ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
+ALERT_HELPER="${ALERT_HELPER:-${REMOTE_DIR}/scripts/send-wecom-group-alert.py}"
 BACKUP_RUN_AT="${BACKUP_RUN_AT:-02:30}"
 WATCHDOG_RUN_AT="${WATCHDOG_RUN_AT:-07:10}"
 SERVICE_PREFIX="${SERVICE_PREFIX:-hyb-oa-automatic-backup}"
@@ -66,7 +67,8 @@ paths_overlap() {
 validate_inputs() {
  local name value
  for name in REMOTE_DIR REMOTE_ENV_FILE POSTGRES_DATA_DIR STORAGE_HOST_DIR \
-    AUTOMATIC_BACKUP_DIR AUTOMATIC_BACKUP_STATE_DIR OFFHOST_SSH_KEY OFFHOST_KNOWN_HOSTS; do
+    AUTOMATIC_BACKUP_DIR AUTOMATIC_BACKUP_STATE_DIR OFFHOST_SSH_KEY OFFHOST_KNOWN_HOSTS \
+    ALERT_WEBHOOK_FILE ALERT_HELPER; do
    value="${!name:-}"
    validate_absolute_path "$name" "$value"
  done
@@ -91,10 +93,14 @@ validate_inputs() {
    && "$(stat -c %a "$OFFHOST_SSH_KEY")" == "600" ]] \
    || fail "off-host private key must exist with mode 0600"
  [[ -s "$OFFHOST_KNOWN_HOSTS" ]] || fail "pinned known-hosts file is missing"
+  [[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
+    && "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" \
+    && "$(stat -c %u "$ALERT_WEBHOOK_FILE")" == "0" ]] \
+    || fail "alert webhook file must be root-owned with mode 0600"
+  [[ -f "$ALERT_HELPER" && ! -L "$ALERT_HELPER" ]] || fail "alert helper is missing"
  [[ "$BACKUP_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "backup time is invalid"
  [[ "$WATCHDOG_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "watchdog time is invalid"
  [[ "$SERVICE_PREFIX" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "service prefix is invalid"
-  [[ "$ALERT_RECIPIENT_EMPLOYEE_NO" =~ ^[A-Za-z0-9_-]{1,80}$ ]] || fail "alert recipient id is invalid"
  local -a isolated_paths=(
    "$POSTGRES_DATA_DIR" "$STORAGE_HOST_DIR" "$AUTOMATIC_BACKUP_DIR" "$AUTOMATIC_BACKUP_STATE_DIR"
  )
@@ -123,7 +129,8 @@ Environment=AUTOMATIC_BACKUP_STATE_DIR=${AUTOMATIC_BACKUP_STATE_DIR}
 Environment=OFFHOST_TARGET=${OFFHOST_TARGET}
 Environment=OFFHOST_SSH_KEY=${OFFHOST_SSH_KEY}
 Environment=OFFHOST_KNOWN_HOSTS=${OFFHOST_KNOWN_HOSTS}
-Environment=ALERT_RECIPIENT_EMPLOYEE_NO=${ALERT_RECIPIENT_EMPLOYEE_NO}
+Environment=ALERT_WEBHOOK_FILE=${ALERT_WEBHOOK_FILE}
+Environment=ALERT_HELPER=${ALERT_HELPER}
 ENVIRONMENT
 }

@@ -7,6 +7,15 @@ PUBLIC_KEY_FILE="${PUBLIC_KEY_FILE:-}"
 RECEIVER_SOURCE="${RECEIVER_SOURCE:-}"
 RECEIVER_TARGET="${RECEIVER_TARGET:-/usr/local/sbin/hyb-oa-offhost-receiver}"
 CONFIG_FILE="${CONFIG_FILE:-/etc/hyb-oa-offhost-backup.conf}"
+ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
+ALERT_HELPER_SOURCE="${ALERT_HELPER_SOURCE:-}"
+ALERT_HELPER_TARGET="${ALERT_HELPER_TARGET:-/usr/local/sbin/hyb-oa-wecom-group-alert}"
+DEADMAN_SOURCE="${DEADMAN_SOURCE:-}"
+DEADMAN_TARGET="${DEADMAN_TARGET:-/usr/local/sbin/hyb-oa-offhost-deadman}"
+DEADMAN_MAX_AGE_SECONDS="${DEADMAN_MAX_AGE_SECONDS:-93600}"
+DEADMAN_RUN_AT="${DEADMAN_RUN_AT:-06:30}"
+DEADMAN_SERVICE="${DEADMAN_SERVICE:-hyb-oa-offhost-deadman}"
+SYSTEMD_DIR="${SYSTEMD_DIR:-/etc/systemd/system}"
 DAILY_KEEP="${DAILY_KEEP:-7}"
 WEEKLY_KEEP="${WEEKLY_KEEP:-4}"
 MODE="${1:-install}"
@@ -45,7 +54,7 @@ validate_backup_root() {
  if [[ -d "$BACKUP_ROOT" ]]; then
    local unmanaged_entry
    unmanaged_entry="$(find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 \
-      ! -name inbox ! -name archive ! -name weekly -print -quit)"
+      ! -name inbox ! -name archive ! -name weekly ! -name state -print -quit)"
    [[ -z "$unmanaged_entry" ]] || fail "backup root contains an unmanaged top-level entry"
  fi
 }
@@ -57,8 +66,22 @@ validate_inputs() {
  validate_absolute_path RECEIVER_SOURCE "$RECEIVER_SOURCE"
  validate_absolute_path RECEIVER_TARGET "$RECEIVER_TARGET"
  validate_absolute_path CONFIG_FILE "$CONFIG_FILE"
+  validate_absolute_path ALERT_WEBHOOK_FILE "$ALERT_WEBHOOK_FILE"
+  validate_absolute_path ALERT_HELPER_SOURCE "$ALERT_HELPER_SOURCE"
+  validate_absolute_path ALERT_HELPER_TARGET "$ALERT_HELPER_TARGET"
+  validate_absolute_path DEADMAN_SOURCE "$DEADMAN_SOURCE"
+  validate_absolute_path DEADMAN_TARGET "$DEADMAN_TARGET"
  [[ -f "$PUBLIC_KEY_FILE" && ! -L "$PUBLIC_KEY_FILE" ]] || fail "public key file is missing"
  [[ -f "$RECEIVER_SOURCE" && ! -L "$RECEIVER_SOURCE" ]] || fail "receiver source is missing"
+  [[ -f "$ALERT_HELPER_SOURCE" && ! -L "$ALERT_HELPER_SOURCE" ]] || fail "alert helper source is missing"
+  [[ -f "$DEADMAN_SOURCE" && ! -L "$DEADMAN_SOURCE" ]] || fail "dead-man source is missing"
+  [[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
+    && "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" \
+    && "$(stat -c %u "$ALERT_WEBHOOK_FILE")" == "0" ]] \
+    || fail "alert webhook file must be root-owned with mode 0600"
+  [[ "$DEADMAN_MAX_AGE_SECONDS" =~ ^[1-9][0-9]*$ ]] || fail "dead-man threshold must be positive"
+  [[ "$DEADMAN_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "dead-man time is invalid"
+  [[ "$DEADMAN_SERVICE" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "dead-man service name is invalid"
  [[ "$DAILY_KEEP" =~ ^[1-9][0-9]*$ && "$WEEKLY_KEEP" =~ ^[1-9][0-9]*$ ]] \
    || fail "retention values must be positive"
  command -v rsync >/dev/null 2>&1 || fail "rsync must be installed before receiver setup"
@@ -86,22 +109,28 @@ install_receiver() {
  fi
  passwd -l "$BACKUP_USER" >/dev/null 2>&1 || true

-  local home_dir inbox_dir archive_dir weekly_dir ssh_dir auth_file public_key rrsync_bin
+  local home_dir inbox_dir archive_dir weekly_dir state_dir ssh_dir auth_file public_key rrsync_bin
  home_dir="$(getent passwd "$BACKUP_USER" | awk -F: '{print $6}')"
  validate_absolute_path BACKUP_USER_HOME "$home_dir"
  inbox_dir="$BACKUP_ROOT/inbox"
  archive_dir="$BACKUP_ROOT/archive"
  weekly_dir="$BACKUP_ROOT/weekly"
-  mkdir -p "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh"
+  state_dir="$BACKUP_ROOT/state"
+  mkdir -p "$inbox_dir" "$archive_dir" "$weekly_dir" "$state_dir" "$home_dir/.ssh"
  [[ -d "$BACKUP_ROOT" && ! -L "$BACKUP_ROOT" \
    && -d "$inbox_dir" && ! -L "$inbox_dir" \
    && -d "$archive_dir" && ! -L "$archive_dir" \
    && -d "$weekly_dir" && ! -L "$weekly_dir" ]] \
    || fail "backup receiver directories must not be symbolic links"
-  chown -R "$BACKUP_USER:$BACKUP_USER" "$BACKUP_ROOT" "$home_dir/.ssh"
+  chown "$BACKUP_USER:$BACKUP_USER" "$BACKUP_ROOT"
+  chown -R "$BACKUP_USER:$BACKUP_USER" "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh"
  chmod 0700 "$BACKUP_ROOT" "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh"
+  chown -R root:root "$state_dir"
+  chmod 0700 "$state_dir"

  install -o root -g root -m 0755 "$RECEIVER_SOURCE" "$RECEIVER_TARGET"
+  install -o root -g root -m 0755 "$ALERT_HELPER_SOURCE" "$ALERT_HELPER_TARGET"
+  install -o root -g root -m 0755 "$DEADMAN_SOURCE" "$DEADMAN_TARGET"
  rrsync_bin="$(command -v rrsync)"
  cat >"$CONFIG_FILE" <<CONFIG
 BACKUP_INBOX_DIR=${inbox_dir}
@@ -130,7 +159,37 @@ AUTH

  runuser -u "$BACKUP_USER" -- env RECEIVER_CONFIG_FILE="$CONFIG_FILE" \
    "$RECEIVER_TARGET" validate-config >/dev/null
-  echo "[offhost-receiver-installer][PASS] restricted receiver installed"
+  cat >"$SYSTEMD_DIR/${DEADMAN_SERVICE}.service" <<SERVICE
+[Unit]
+Description=hyb-oa independent off-host backup dead-man
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+UMask=0077
+ExecStart=/usr/bin/python3 ${DEADMAN_TARGET} --archive-root ${archive_dir} --state-dir ${state_dir} --webhook-file ${ALERT_WEBHOOK_FILE} --alert-helper ${ALERT_HELPER_TARGET} --max-age-seconds ${DEADMAN_MAX_AGE_SECONDS}
+SERVICE
+  cat >"$SYSTEMD_DIR/${DEADMAN_SERVICE}.timer" <<TIMER
+[Unit]
+Description=Check hyb-oa off-host backup freshness independently
+
+[Timer]
+OnCalendar=*-*-* ${DEADMAN_RUN_AT}:00
+Persistent=true
+Unit=${DEADMAN_SERVICE}.service
+
+[Install]
+WantedBy=timers.target
+TIMER
+  systemctl daemon-reload
+  echo "[offhost-receiver-installer][PASS] restricted receiver and disabled dead-man installed"
+}
+
+enable_receiver() {
+  install_receiver
+  systemctl enable --now "${DEADMAN_SERVICE}.timer"
+  echo "[offhost-receiver-installer][PASS] dead-man timer enabled"
 }

 uninstall_receiver() {
@@ -138,6 +197,12 @@ uninstall_receiver() {
  [[ "$BACKUP_USER" =~ ^[a-z_][a-z0-9_-]{0,30}$ ]] || fail "backup user is invalid"
  validate_absolute_path RECEIVER_TARGET "$RECEIVER_TARGET"
  validate_absolute_path CONFIG_FILE "$CONFIG_FILE"
+  validate_absolute_path ALERT_HELPER_TARGET "$ALERT_HELPER_TARGET"
+  validate_absolute_path DEADMAN_TARGET "$DEADMAN_TARGET"
+  [[ "$DEADMAN_SERVICE" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "dead-man service name is invalid"
+  systemctl disable --now "${DEADMAN_SERVICE}.timer" >/dev/null 2>&1 || true
+  systemctl stop "${DEADMAN_SERVICE}.service" >/dev/null 2>&1 || true
+  rm -f -- "$SYSTEMD_DIR/${DEADMAN_SERVICE}.timer" "$SYSTEMD_DIR/${DEADMAN_SERVICE}.service"
  if id "$BACKUP_USER" >/dev/null 2>&1; then
    local home_dir auth_file
    home_dir="$(getent passwd "$BACKUP_USER" | awk -F: '{print $6}')"
@@ -146,19 +211,23 @@ uninstall_receiver() {
      rm -f -- "$auth_file"
    fi
  fi
-  rm -f -- "$RECEIVER_TARGET" "$CONFIG_FILE"
-  echo "[offhost-receiver-installer][PASS] receiver authorization removed; archives retained"
+  rm -f -- "$RECEIVER_TARGET" "$CONFIG_FILE" "$ALERT_HELPER_TARGET" "$DEADMAN_TARGET"
+  systemctl daemon-reload
+  echo "[offhost-receiver-installer][PASS] receiver authorization and dead-man removed; archives retained"
 }

 case "$MODE" in
  install)
    install_receiver
    ;;
+  enable)
+    enable_receiver
+    ;;
  uninstall|--uninstall)
    uninstall_receiver
    ;;
  --help|-h)
-    echo "Usage: BACKUP_ROOT=<absolute> PUBLIC_KEY_FILE=<absolute> RECEIVER_SOURCE=<absolute> bash install-offhost-backup-receiver.sh [install|uninstall]"
+    echo "Usage: <explicit backup/key/helper/dead-man/webhook paths> bash install-offhost-backup-receiver.sh [install|enable|uninstall]"
    ;;
  *)
    fail "unsupported mode"
@@ -226,29 +226,46 @@ manifest 必须至少记录一个真实来源标识：活动 `/ready` 有构建

 整批复制到获批的异机/异盘位置并记录保留周期；不能只复制 DB 或附件中的一份。

-#### 每日自动异机备份（当前仅本地实现，远端启用为 EXTERNAL_GATED）
+#### 每日自动异机备份（可靠启用闭环仍为 EXTERNAL_GATED）

 - 每日 02:30 由生产主机解析唯一活动 Compose project 和真实 DB/附件挂载，生成同批三文件，ARM 校验并原子归档成功后才写成功状态和执行源端轮转。
- 生产保留 7 个自动日批次；ARM 保留 7 个日批次和 4 个周检查点。07:10 watchdog 检查最近 24 小时异机成功记录；季度 timer 只发送人工影子恢复提醒。
+- 生产保留 7 个自动日批次；ARM 保留 7 个日批次和 4 个周检查点。07:10 源端 watchdog 检查最近 24 小时异机成功记录；ARM 独立 dead-man 检查最近 26 小时归档，生产整机或 OA server 停止时仍能告警；季度 timer 只发送人工影子恢复提醒。
 - ARM 使用专用密码锁定账号和 forced-command key：rsync 只能写 inbox，不能读取/删除 archive，也没有普通 shell。生产固定专用 key 和人工核对的 known-hosts，禁止 `accept-new`。
- 失败告警只包含阶段、状态、finding 数和类别建议。同一失败降噪；首次失败、状态恶化、新 finding 和每季度首次恢复提醒才发送。
+- 同一 UTC 日重试使用新 run id 时，ARM 必须先完整校验候选，再原子取代当日旧批次并重建该周硬链接检查点；坏候选或发布失败不得触碰原有效批次。
+- 生产与 ARM 都直接使用独立企微运维群机器人，不依赖 OA server、数据库或个人工号。Webhook 正文只由用户手工写入两台主机的 root-only `0600` 文件；仓库、普通 `.env`、systemd unit、日志和聊天只能出现受控文件路径，不能出现 URL 或 key。
+- 失败告警只包含阶段、状态、finding 数和类别建议；原始命令诊断只保留在源端状态目录的 `0600` 最近失败日志。同一失败降噪；首次失败、状态恶化、新 finding、恢复通知和每季度首次恢复提醒才发送。新部署构建与最近恢复点 build SHA 不同只记 finding，缺失、损坏、超龄、未来时间或 Compose project 不同仍硬失败。
+
+创建机器人后，由用户分别登录两台主机，在 root shell 中用静默输入把完整 Webhook 写入受控路径；不要把值作为命令参数、环境文件内容或聊天消息交给 Codex：
+
+```bash
+umask 077
+read -rsp 'WeCom operations robot Webhook: ' HYB_OA_ALERT_WEBHOOK
+printf '\n'
+install -o root -g root -m 0600 /dev/null <controlled-webhook-file>
+printf '%s\n' "$HYB_OA_ALERT_WEBHOOK" > <controlled-webhook-file>
+unset HYB_OA_ALERT_WEBHOOK
+```

 安装顺序必须是：先获批 ARM 端安装 rsync、受限身份和 receiver，用合成批次验证；再获批生产端安装 units，手工完成首个真实批次；最后显式执行生产安装器的 `enable`。两个安装器默认 `install` 均不代表已启用，不得把本地合同通过写成远端完成。

 ```bash
 # ARM：参数从受控配置取得；install 会写账号/authorized_keys/receiver，需单独批准。
 BACKUP_ROOT=<absolute-backup-root> PUBLIC_KEY_FILE=<dedicated-public-key> \
-RECEIVER_SOURCE=<repo-receiver-script> \
+RECEIVER_SOURCE=<repo-receiver-script> ALERT_WEBHOOK_FILE=<controlled-webhook-file> \
+ALERT_HELPER_SOURCE=<repo-group-alert-helper> DEADMAN_SOURCE=<repo-deadman-script> \
  bash deploy/trialrun/install-offhost-backup-receiver.sh install

 # 生产：先只安装 disabled units；路径、项目、构建和 SSH 身份必须绑定活动实例。
-<explicit-project-data-build-and-ssh-env> \
+<explicit-project-data-build-ssh-and-webhook-file-env> \
  bash deploy/trialrun/install-automatic-backup-timer.sh install

 # 首批真实备份、ARM 哈希和脱敏告警均经单独确认后，才允许启用三组 timer。
+<same-arm-explicit-env> bash deploy/trialrun/install-offhost-backup-receiver.sh enable
 <same-explicit-env> bash deploy/trialrun/install-automatic-backup-timer.sh enable
 ```

+`BACKUP_RUN_AT` 使用生产主机本地时区，ARM 归档按 UTC 日分桶。调整执行时刻的当天可能出现预期空窗；改动后必须人工确认次日 UTC 分桶、下一次 daily 运行和 26 小时 dead-man 窗口。真实告警验收必须真正停止一次源端 timer 或故意让一次传输失败，再确认 ARM 首次告警、重复降噪和恢复通知；只测试机器人能发消息不算通过。
+
 卸载只撤销 timer/receiver 授权，保留源端批次、状态、ARM archive 和凭证文件；专用账号/密钥的最终销毁另走获批清理。

 ### 1.5.2 本地隔离恢复演练
@@ -284,7 +301,7 @@ RESTORE_SCOPE=shadow \

 `restore-db.sh` 和 `restore-storage.sh` 默认只接受 `shadow`。影子库名必须包含独立的 `shadow`、`drill` 或 `restore_test` 标记；数据库确认不匹配、库名不安全、附件目录非空、目标为默认试运行附件目录或归档路径危险时均在写入前退出。

-真实格式影子恢复的验收不仅是 `/ready`：恢复后的 `_prisma_migrations` 成功项必须等于冻结基线（当前预期 58），预先列出的关键业务表计数必须逐表一致，附件解包后的普通文件数必须与活动源目录一致。任一项不一致即判定批次不可作为切换回滚点。
+真实格式影子恢复的验收不仅是 `/ready`：恢复后的 `_prisma_migrations` 成功项必须等于执行前现场冻结基线（当前生产证据为 61 项；本地 62 项仅表示下一次部署预计新增 1 项），预先列出的关键业务表计数必须逐表一致，附件解包后的普通文件数必须与活动源目录一致。任一项不一致即判定批次不可作为切换回滚点。

 ### 1.5.3 生产切换前冻结记录

@@ -1,6 +1,7 @@
 #!/usr/bin/env node
 const fs = require('node:fs');
 const path = require('node:path');
+const childProcess = require('node:child_process');

 const rootDir = path.resolve(__dirname, '..', '..');
 const configPath = path.join(rootDir, 'server/src/config.js');
@@ -29,6 +30,8 @@ const offhostReceiverInstallerPath = path.join(
 );
 const automaticBackupRunnerPath = path.join(rootDir, 'scripts/run-automatic-backup.sh');
 const offhostReceiverPath = path.join(rootDir, 'scripts/receive-offhost-backup.sh');
+const groupAlertHelperPath = path.join(rootDir, 'scripts/send-wecom-group-alert.py');
+const offhostDeadmanPath = path.join(rootDir, 'scripts/check-offhost-backup-deadman.py');
 const nginxPaths = [
  path.join(rootDir, 'deploy/trialrun/nginx-hyb-oa.conf'),
  path.join(rootDir, 'deploy/trialrun/nginx.conf.example'),
@@ -157,6 +160,8 @@ const automaticBackupTimerScriptContent = readFile(automaticBackupTimerScriptPat
 const offhostReceiverInstallerContent = readFile(offhostReceiverInstallerPath);
 const automaticBackupRunnerContent = readFile(automaticBackupRunnerPath);
 const offhostReceiverContent = readFile(offhostReceiverPath);
+const groupAlertHelperContent = readFile(groupAlertHelperPath);
+const offhostDeadmanContent = readFile(offhostDeadmanPath);
 const nginxContents = nginxPaths.map((filePath) => ({
  filePath,
  content: readFile(filePath),
@@ -185,6 +190,25 @@ for (const key of ['APP_HOST_PORT', 'POSTGRES_DATA_DIR', 'STORAGE_HOST_DIR']) {
 if (envExampleKeys.has('COMPOSE_PROJECT_NAME')) {
  errors.push('.env.example 不应持久化 COMPOSE_PROJECT_NAME，调用方必须通过 -p 显式选择项目');
 }
+for (const key of ['ALERT_WEBHOOK_FILE', 'WECOM_WEBHOOK_URL', 'WECOM_WEBHOOK_KEY']) {
+  if (envExampleKeys.has(key)) {
+    errors.push(`.env.example 不应持久化企微群机器人配置 ${key}`);
+  }
+}
+const trackedFiles = childProcess.execFileSync('git', ['ls-files', '--cached', '--others', '--exclude-standard', '-z'], {
+  cwd: rootDir,
+  encoding: 'utf8',
+}).split('\0').filter(Boolean);
+const literalWebhookPattern = /https:\/\/qyapi\.weixin\.qq\.com\/cgi-bin\/webhook\/send\?key=/;
+for (const relativePath of trackedFiles) {
+  const filePath = path.join(rootDir, relativePath);
+  if (!fs.statSync(filePath).isFile()) continue;
+  const content = fs.readFileSync(filePath);
+  if (content.includes(0)) continue;
+  if (literalWebhookPattern.test(content.toString('utf8'))) {
+    errors.push(`${relativePath} 不得持久化完整企微群机器人 Webhook URL`);
+  }
+}

 for (const serviceName of requiredServices) {
  const block = extractServiceBlock(composeContent, serviceName);
@@ -306,6 +330,8 @@ for (const requiredFragment of [
  'Environment=OFFHOST_TARGET=${OFFHOST_TARGET}',
  'Environment=OFFHOST_SSH_KEY=${OFFHOST_SSH_KEY}',
  'Environment=OFFHOST_KNOWN_HOSTS=${OFFHOST_KNOWN_HOSTS}',
+  'Environment=ALERT_WEBHOOK_FILE=${ALERT_WEBHOOK_FILE}',
+  'Environment=ALERT_HELPER=${ALERT_HELPER}',
  'install)\n    install_units',
  'enable)\n    enable_units',
 ]) {
@@ -325,7 +351,10 @@ for (const requiredFragment of [
  'hyb-backup-finalize $run_id $manifest_sha',
  'write_success_state "$run_id" "$manifest_sha"',
  'prune_local_backups "$batch_name"',
-  '--summary-file=-',
+  'python3 "$ALERT_HELPER"',
+  '--webhook-file "$ALERT_WEBHOOK_FILE"',
+  'capture_command offhost_transfer rsync',
+  'last-error.log',
  "docker inspect -f '{{json .Mounts}}'",
  'if len(matches) != 1 or not matches[0]:',
 ]) {
@@ -345,6 +374,8 @@ for (const requiredFragment of [
  'batch must contain exactly three regular files',
  'manifest artifact checksum mismatch',
  'mv -- "$partial_dir" "$archive_dir"',
+  'replacement_rollback=',
+  'publish_weekly_checkpoint',
  'prune_named_directories "$BACKUP_ARCHIVE_DIR"',
  'prune_named_directories "$BACKUP_WEEKLY_DIR"',
 ]) {
@@ -360,6 +391,9 @@ for (const requiredFragment of [
  'BACKUP_INBOX_DIR=${inbox_dir}',
  'BACKUP_ARCHIVE_DIR=${archive_dir}',
  'BACKUP_WEEKLY_DIR=${weekly_dir}',
+  'DEADMAN_MAX_AGE_SECONDS="${DEADMAN_MAX_AGE_SECONDS:-93600}"',
+  'ExecStart=/usr/bin/python3 ${DEADMAN_TARGET}',
+  'systemctl enable --now "${DEADMAN_SERVICE}.timer"',
  'archives retained',
  'backup root contains an unmanaged top-level entry',
  'backup root must not be inside a system-managed directory',
@@ -369,6 +403,30 @@ for (const requiredFragment of [
  }
 }

+for (const requiredFragment of [
+  'stat.S_IMODE(info.st_mode) != 0o600',
+  'parsed.hostname != "qyapi.weixin.qq.com"',
+  'HYB_WECOM_ALERT_TEST_MODE',
+  'should_send(previous, status, category)',
+  '不要在群内粘贴原始日志',
+]) {
+  if (!groupAlertHelperContent.includes(requiredFragment)) {
+    errors.push(`send-wecom-group-alert.py 缺少独立脱敏告警合同: ${requiredFragment}`);
+  }
+}
+
+for (const requiredFragment of [
+  'default=93600',
+  'archive_manifest_invalid',
+  'archive_time_in_future',
+  'FAIL_OFFHOST_DEADMAN',
+  'deadman-alert-state.json',
+]) {
+  if (!offhostDeadmanContent.includes(requiredFragment)) {
+    errors.push(`check-offhost-backup-deadman.py 缺少 ARM 独立看门狗合同: ${requiredFragment}`);
+  }
+}
+
 for (const requiredFragment of [
  'validate_data_path POSTGRES_DATA_DIR "$POSTGRES_DATA_DIR"',
  'validate_data_path STORAGE_HOST_DIR "$STORAGE_HOST_DIR"',
@@ -359,12 +359,13 @@ WAPC2 请假/补卡只读 POC 操作口径：
 自动异机备份运维口径：

 1. 自动备份以 OA 仓库 runbook 首部“紧急恢复卡”为唯一短入口；公司知识库保存该卡副本，并指定系统管理员和一名替补操作人。
-2. 每日 02:30 生成数据库、附件、manifest 同批恢复点并推送 ARM；07:10 检查最近 24 小时异机成功状态。生产保留 7 个自动日批次，ARM 保留 7 个日批次和 4 个周检查点。
+2. 每日 02:30 生成数据库、附件、manifest 同批恢复点并推送 ARM；07:10 检查最近 24 小时异机成功状态，ARM 另以 26 小时 dead-man 独立检查归档。生产保留 7 个自动日批次，ARM 保留 7 个日批次和 4 个周检查点。
 3. ARM 专用身份只能写 inbox 和请求 finalize，不能登录 shell、读取或删除 archive。任何人不得为了省事改用普通 ARM 登录账号或关闭 known-hosts 固定校验。
-4. 企微只收脱敏异常：首次失败、恶化或新 finding 发送；同一失败降噪。季度提醒只要求人工在 ARM 隔离目标恢复，不会自动恢复生产。
-5. 当前仓库实现和本地合成合同不等于服务器已安装。ARM receiver、生产 units、首次真实批次、timer enable 和真实失败告警均为 **EXTERNAL_GATED**，必须逐阶段批准并回填证据。
+4. 自动备份使用独立企微运维群机器人，不依赖 OA server 或个人员工号。Webhook 只由用户写入两台主机的 root-only 文件，不进仓库、普通 `.env`、日志或聊天。企微只收脱敏异常、finding 和恢复通知；同一失败降噪。季度提醒只要求人工在 ARM 隔离目标恢复，不会自动恢复生产。
+5. 当前 receiver 和生产 units 的既有安装不等于可靠启用闭环完成。群机器人、ARM dead-man、更新后的两端脚本、首次有效真实批次、全部 timer enable 和真实断链告警均为 **EXTERNAL_GATED**，必须逐阶段批准并回填证据；真实验收必须停止一次源端 timer 或故意让一次传输失败，不能只测试机器人发送。
 6. 每季度由替补操作人按恢复卡走查，至少能指出归档类别、三条恢复入口、`/ready` 和关键计数验收项。不能独立说明时，本季度恢复门禁不通过。
 7. 卸载 timer 或 receiver 时保留历史 archive 和源端完整批次；账号、密钥、备份数据的删除必须另行批准。
+8. 改动每日备份时刻后，人工确认次日 UTC 分桶和 26 小时窗口，避免把调整日的预期空窗误当成长期故障。

 WROAD2/WARCH3 企微受控上线每日巡检：

@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+"""Validate the newest ARM archive and drive an independent dead-man alert."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import stat
+import subprocess
+import sys
+import tempfile
+from datetime import datetime, timezone
+from pathlib import Path
+
+
+BATCH_RE = re.compile(r"^hyb-oa-backup-(auto-(\d{8}T\d{6}Z))$")
+
+
+def fail(message: str) -> "NoReturn":
+    print(f"[offhost-deadman][ERROR] {message}", file=sys.stderr)
+    raise SystemExit(1)
+
+
+def sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def validate_latest(archive_root: Path, max_age: int) -> tuple[str, str]:
+    if not archive_root.is_dir() or archive_root.is_symlink():
+        return "FAIL_OFFHOST_DEADMAN", "archive_root_invalid"
+    candidates = sorted(
+        (entry for entry in archive_root.iterdir() if entry.is_dir() and not entry.is_symlink() and BATCH_RE.fullmatch(entry.name)),
+        key=lambda entry: entry.name,
+    )
+    if not candidates:
+        return "FAIL_OFFHOST_DEADMAN", "archive_missing"
+    batch = candidates[-1]
+    match = BATCH_RE.fullmatch(batch.name)
+    assert match is not None
+    run_id, stamp = match.groups()
+    expected_names = {"database.dump", "storage.tar.gz", "manifest.json"}
+    entries = list(batch.iterdir())
+    if {entry.name for entry in entries} != expected_names or any(
+        entry.is_symlink() or not stat.S_ISREG(entry.stat().st_mode) for entry in entries
+    ):
+        return "FAIL_OFFHOST_DEADMAN", "archive_files_invalid"
+    try:
+        manifest = json.loads((batch / "manifest.json").read_text(encoding="utf-8"))
+        if manifest.get("schemaVersion") != 1 or manifest.get("status") != "complete" or manifest.get("runId") != run_id:
+            raise ValueError
+        artifacts = manifest.get("artifacts") or {}
+        for key, name in (("database", "database.dump"), ("storage", "storage.tar.gz")):
+            item = artifacts.get(key) or {}
+            target = batch / name
+            if item.get("file") != name or item.get("bytes") != target.stat().st_size or item.get("sha256") != sha256(target):
+                raise ValueError
+        completed = datetime.strptime(stamp, "%Y%m%dT%H%M%SZ").replace(tzinfo=timezone.utc)
+    except (OSError, ValueError, TypeError, json.JSONDecodeError):
+        return "FAIL_OFFHOST_DEADMAN", "archive_manifest_invalid"
+    age = (datetime.now(timezone.utc) - completed).total_seconds()
+    if age < 0:
+        return "FAIL_OFFHOST_DEADMAN", "archive_time_in_future"
+    if age > max_age:
+        return "FAIL_OFFHOST_DEADMAN", "archive_stale"
+    return "PASS_OFFHOST_DEADMAN_FRESH", "fresh_offhost_archive"
+
+
+def write_summary(path: Path, status: str, category: str) -> None:
+    finding = status.startswith("FAIL_")
+    payload = {
+        "issueId": "AUTO_BACKUP",
+        "status": status,
+        "conclusion": status,
+        "findingCount": 1 if finding else 0,
+        "findings": ([{"source": "offhost_deadman", "status": status, "severity": "P0", "detail": category}] if finding else []),
+        "privacy": "status/count/category only; no credentials, host addresses, absolute paths, file names, or payloads",
+    }
+    path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
+    fd, temporary = tempfile.mkstemp(prefix=".deadman-summary.", dir=path.parent)
+    with os.fdopen(fd, "w", encoding="utf-8") as handle:
+        json.dump(payload, handle, ensure_ascii=True, indent=2)
+        handle.write("\n")
+    os.chmod(temporary, 0o600)
+    os.replace(temporary, path)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--archive-root", required=True, type=Path)
+    parser.add_argument("--state-dir", required=True, type=Path)
+    parser.add_argument("--webhook-file", required=True, type=Path)
+    parser.add_argument("--alert-helper", required=True, type=Path)
+    parser.add_argument("--max-age-seconds", type=int, default=93600)
+    args = parser.parse_args()
+    if args.max_age_seconds <= 0:
+        fail("max age must be positive")
+    try:
+        args.state_dir.mkdir(parents=True, exist_ok=True, mode=0o700)
+        if args.state_dir.is_symlink() or not args.state_dir.is_dir():
+            fail("state directory is invalid")
+        os.chmod(args.state_dir, 0o700)
+        status, category = validate_latest(args.archive_root, args.max_age_seconds)
+        summary = args.state_dir / "last-deadman.json"
+        write_summary(summary, status, category)
+        result = subprocess.run(
+            [sys.executable, str(args.alert_helper), "--webhook-file", str(args.webhook_file), "--summary-file", str(summary), "--state-file", str(args.state_dir / "deadman-alert-state.json")],
+            check=False,
+        )
+    except OSError:
+        fail("dead-man local file operation failed")
+    if result.returncode != 0:
+        fail("alert evaluation failed")
+    if status.startswith("FAIL_"):
+        fail(category)
+    print("[offhost-deadman][PASS] newest archive is fresh and valid")
+
+
+if __name__ == "__main__":
+    main()
@@ -78,7 +78,7 @@ receiver() {
    "$ROOT_DIR/scripts/receive-offhost-backup.sh" "$@"
 }

-CURRENT_RUN_ID="auto-$(date -u +%Y%m%dT%H%M%SZ)"
+CURRENT_RUN_ID="auto-$(date -u +%Y%m%d)T000000Z"
 CURRENT_BATCH="hyb-oa-backup-$CURRENT_RUN_ID"

 VALID_LAYOUT="$TEST_DIR/receiver-valid"
@@ -92,15 +92,41 @@ pass "receiver finalizes one complete synthetic batch"
 receiver "$VALID_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$VALID_MANIFEST_SHA" >/dev/null
 pass "receiver treats an identical repeated finalize as idempotent"

-SECOND_RUN_ID="auto-$(date -u +%Y%m%d)T000000Z"
-if [[ "$SECOND_RUN_ID" == "$CURRENT_RUN_ID" ]]; then
-  SECOND_RUN_ID="auto-$(date -u +%Y%m%d)T000001Z"
-fi
+SECOND_RUN_ID="auto-$(date -u +%Y%m%d)T000001Z"
 SECOND_BATCH="hyb-oa-backup-$SECOND_RUN_ID"
 make_batch "$VALID_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID"
 SECOND_MANIFEST_SHA="$(sha256sum "$VALID_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')"
-expect_failure "receiver rejects a different automatic batch for the same UTC day" \
-  receiver "$VALID_LAYOUT" finalize-test "$SECOND_RUN_ID" "$SECOND_MANIFEST_SHA"
+receiver "$VALID_LAYOUT" finalize-test "$SECOND_RUN_ID" "$SECOND_MANIFEST_SHA" >/dev/null
+[[ ! -e "$VALID_LAYOUT/archive/$CURRENT_BATCH" && -d "$VALID_LAYOUT/archive/$SECOND_BATCH" ]]
+WEEKLY_BATCH="$(find "$VALID_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -print -quit)"
+[[ -n "$WEEKLY_BATCH" ]]
+[[ "$(stat -c %i "$WEEKLY_BATCH/database.dump")" == "$(stat -c %i "$VALID_LAYOUT/archive/$SECOND_BATCH/database.dump")" ]]
+pass "receiver atomically replaces a different batch for the same UTC day and rebuilds its weekly checkpoint"
+
+THIRD_RUN_ID="auto-$(date -u +%Y%m%d)T000002Z"
+THIRD_BATCH="hyb-oa-backup-$THIRD_RUN_ID"
+make_batch "$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID"
+printf 'corrupt candidate\n' >>"$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial/database.dump"
+THIRD_MANIFEST_SHA="$(sha256sum "$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')"
+expect_failure "receiver validates a same-day replacement before touching the current archive" \
+  receiver "$VALID_LAYOUT" finalize-test "$THIRD_RUN_ID" "$THIRD_MANIFEST_SHA"
+[[ -d "$VALID_LAYOUT/archive/$SECOND_BATCH" && ! -e "$VALID_LAYOUT/archive/$THIRD_BATCH" ]]
+pass "receiver preserves the current daily archive when a replacement candidate is corrupt"
+
+ROLLBACK_LAYOUT="$TEST_DIR/receiver-replacement-rollback"
+make_layout "$ROLLBACK_LAYOUT"
+make_batch "$ROLLBACK_LAYOUT/inbox/${CURRENT_BATCH}.partial" "$CURRENT_RUN_ID"
+ROLLBACK_FIRST_SHA="$(sha256sum "$ROLLBACK_LAYOUT/inbox/${CURRENT_BATCH}.partial/manifest.json" | awk '{print $1}')"
+receiver "$ROLLBACK_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$ROLLBACK_FIRST_SHA" >/dev/null
+ROLLBACK_WEEKLY="$(find "$ROLLBACK_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -print -quit)"
+find "$ROLLBACK_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -exec rm -rf -- {} +
+printf 'invalid weekly target\n' >"$ROLLBACK_WEEKLY"
+make_batch "$ROLLBACK_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID"
+ROLLBACK_SECOND_SHA="$(sha256sum "$ROLLBACK_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')"
+expect_failure "receiver rolls back the daily archive when weekly checkpoint publication fails" \
+  receiver "$ROLLBACK_LAYOUT" finalize-test "$SECOND_RUN_ID" "$ROLLBACK_SECOND_SHA"
+[[ -d "$ROLLBACK_LAYOUT/archive/$CURRENT_BATCH" && ! -e "$ROLLBACK_LAYOUT/archive/$SECOND_BATCH" ]]
+pass "receiver restores the prior daily point after a later replacement step fails"

 negative_receiver_case() {
  local label="$1"
@@ -225,7 +251,7 @@ if [[ "${1:-}" == "compose" ]]; then
  args=" $* "
  if [[ "$args" == *" ps -q server "* ]]; then echo server-id; exit 0; fi
  if [[ "$args" == *" ps -q db "* ]]; then echo db-id; exit 0; fi
-  if [[ "$args" == *" exec -T server "* ]]; then cat >>"$MOCK_ALERT_LOG"; exit 0; fi
+  if [[ "$args" == *" exec -T server "* ]]; then exit 99; fi
 fi
 if [[ "${1:-}" == "inspect" ]]; then
  template="${3:-}"
@@ -260,6 +286,7 @@ MOCK

 cat >"$MOCK_BIN/rsync" <<'MOCK'
 #!/usr/bin/env bash
+echo 'synthetic transfer diagnostic' >&2
 exit "${MOCK_RSYNC_EXIT:-1}"
 MOCK
 cat >"$MOCK_BIN/ssh" <<'MOCK'
@@ -267,6 +294,34 @@ cat >"$MOCK_BIN/ssh" <<'MOCK'
 exit "${MOCK_SSH_EXIT:-0}"
 MOCK
 chmod +x "$MOCK_BIN/docker" "$MOCK_BIN/rsync" "$MOCK_BIN/ssh"
+WECOM_TEST_HOST='qyapi.weixin.qq.com'
+WECOM_TEST_PATH='/cgi-bin/webhook/send'
+printf 'https://%s%s?key=%s\n' "$WECOM_TEST_HOST" "$WECOM_TEST_PATH" \
+  '00000000-0000-0000-0000-000000000000' >"$TEST_DIR/wecom-webhook"
+chmod 0600 "$TEST_DIR/wecom-webhook"
+ALERT_CAPTURE="$TEST_DIR/alert-capture.jsonl"
+cat >"$TEST_DIR/failure-summary.json" <<'JSON'
+{"issueId":"AUTO_BACKUP","status":"FAIL_AUTOMATIC_BACKUP","conclusion":"FAIL_AUTOMATIC_BACKUP","findingCount":1,"findings":[{"source":"automatic_backup","status":"FAIL_AUTOMATIC_BACKUP","severity":"P0","detail":"synthetic_failure"}]}
+JSON
+chmod 0600 "$TEST_DIR/failure-summary.json"
+chmod 0644 "$TEST_DIR/wecom-webhook"
+expect_failure "group alert helper rejects a webhook file that is not mode 0600" env \
+  HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \
+  python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook" \
+  --summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json"
+chmod 0600 "$TEST_DIR/wecom-webhook"
+ln -s "$TEST_DIR/wecom-webhook" "$TEST_DIR/wecom-webhook-link"
+expect_failure "group alert helper rejects a symbolic-link webhook file" env \
+  HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \
+  python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook-link" \
+  --summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json"
+printf 'https://%s%s?key=%s&extra=forbidden\n' "$WECOM_TEST_HOST" "$WECOM_TEST_PATH" \
+  '00000000-0000-0000-0000-000000000000' >"$TEST_DIR/wecom-webhook-invalid"
+chmod 0600 "$TEST_DIR/wecom-webhook-invalid"
+expect_failure "group alert helper rejects a webhook URL with an unexpected query field" env \
+  HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \
+  python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook-invalid" \
+  --summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json"

 wrapper_env=(
  env PATH="$MOCK_BIN:$PATH" REMOTE_DIR="$WRAPPER_ROOT"
@@ -275,7 +330,9 @@ wrapper_env=(
  STORAGE_HOST_DIR="$WRAPPER_STORAGE" BUILD_SHA="$BUILD_SHA" VITE_BUILD_SHA="$VITE_BUILD_SHA"
  AUTOMATIC_BACKUP_DIR="$WRAPPER_BACKUPS" AUTOMATIC_BACKUP_STATE_DIR="$WRAPPER_STATE"
  OFFHOST_TARGET="hybbackup@backup-alias" OFFHOST_SSH_KEY="$TEST_DIR/offhost-key"
-  OFFHOST_KNOWN_HOSTS="$TEST_DIR/known-hosts" ALERT_RECIPIENT_EMPLOYEE_NO="OPS001"
+  OFFHOST_KNOWN_HOSTS="$TEST_DIR/known-hosts" ALERT_WEBHOOK_FILE="$TEST_DIR/wecom-webhook"
+  ALERT_HELPER="$ROOT_DIR/scripts/send-wecom-group-alert.py"
+  HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE"
 )

 exec 8>"$WRAPPER_STATE/automatic-backup.lock"
@@ -305,6 +362,9 @@ assert payload["status"] == "FAIL_AUTOMATIC_BACKUP"
 assert not any(value in serialized for value in ("/tmp/", "postgresql://", "backup-alias", "fixture"))
 PYTHON
 pass "wrapper failure summary is category-only and redacted"
+[[ -f "$WRAPPER_STATE/last-error.log" && "$(stat -c %a "$WRAPPER_STATE/last-error.log")" == "600" ]]
+grep -Fq 'synthetic transfer diagnostic' "$WRAPPER_STATE/last-error.log"
+pass "wrapper retains the latest command diagnostic locally with mode 0600"

 find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' -exec rm -rf -- {} +
 for day in 01 02 03 04 05 06 07; do
@@ -315,6 +375,8 @@ done
 [[ -f "$WRAPPER_STATE/last-success.json" ]]
 [[ "$(find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' | wc -l)" == "7" ]]
 pass "off-host finalize success writes state and then prunes source retention to seven"
+[[ ! -e "$WRAPPER_STATE/last-error.log" ]]
+pass "wrapper clears the retained diagnostic after a successful off-host run"

 rm -f "$WRAPPER_STATE/last-success.json"
 expect_failure "stale watchdog rejects a missing success record" \
@@ -329,7 +391,7 @@ with open(sys.argv[1], "w", encoding="utf-8") as handle:
        "schemaVersion": 1,
        "status": "complete",
        "completedAt": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),
-        "runId": "auto-synthetic",
+        "runId": "auto-20260727T000000Z",
        "manifestSha256": "0" * 64,
        "composeProject": "hyb-oa-current",
        "buildSha": sys.argv[2],
@@ -345,12 +407,89 @@ payload["buildSha"] = "0" * 40
 with open(sys.argv[1], "w", encoding="utf-8") as handle:
    json.dump(payload, handle)
 PYTHON
-expect_failure "stale watchdog rejects a success record from another build" \
-  "${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
+"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale >/dev/null
+python3 - "$WRAPPER_STATE/last-run.json" <<'PYTHON'
+import json
+import sys
+with open(sys.argv[1], encoding="utf-8") as handle:
+    payload = json.load(handle)
+assert payload["status"] == "PASS_WITH_FINDINGS"
+assert payload["findings"][0]["detail"] == "recovery_point_build_differs"
+PYTHON
+pass "stale watchdog downgrades a fresh recovery point from another build to one finding"
 mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json"
 "${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale >/dev/null
 pass "stale watchdog accepts a fresh project/build-bound success record"

+cp "$WRAPPER_STATE/last-success.json" "$WRAPPER_STATE/last-success.valid.json"
+python3 - "$WRAPPER_STATE/last-success.json" <<'PYTHON'
+import json
+import sys
+with open(sys.argv[1], encoding="utf-8") as handle:
+    payload = json.load(handle)
+payload["composeProject"] = "different-project"
+with open(sys.argv[1], "w", encoding="utf-8") as handle:
+    json.dump(payload, handle)
+PYTHON
+expect_failure "stale watchdog hard-fails a fresh record from another Compose project" \
+  "${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
+mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json"
+cp "$WRAPPER_STATE/last-success.json" "$WRAPPER_STATE/last-success.valid.json"
+python3 - "$WRAPPER_STATE/last-success.json" <<'PYTHON'
+import json
+import sys
+with open(sys.argv[1], encoding="utf-8") as handle:
+    payload = json.load(handle)
+payload["manifestSha256"] = "invalid"
+with open(sys.argv[1], "w", encoding="utf-8") as handle:
+    json.dump(payload, handle)
+PYTHON
+expect_failure "stale watchdog hard-fails a malformed success record" \
+  "${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
+mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json"
+
+DEADMAN_LAYOUT="$TEST_DIR/deadman"
+mkdir -p "$DEADMAN_LAYOUT/archive" "$DEADMAN_LAYOUT/state"
+cp -a "$VALID_LAYOUT/archive/$SECOND_BATCH" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH"
+DEADMAN_CAPTURE="$TEST_DIR/deadman-alerts.jsonl"
+deadman_env=(env HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$DEADMAN_CAPTURE")
+"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
+  --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
+  --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null
+[[ ! -e "$DEADMAN_CAPTURE" ]]
+pass "ARM dead-man accepts a fresh verified archive without sending a healthy baseline"
+DEADMAN_CORRUPT_STATE="$TEST_DIR/deadman-corrupt-state"
+DEADMAN_CORRUPT_CAPTURE="$TEST_DIR/deadman-corrupt-alerts.jsonl"
+mkdir "$DEADMAN_CORRUPT_STATE"
+chmod 0600 "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
+printf 'corruption\n' >>"$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
+expect_failure "ARM dead-man rejects a corrupt newest finalized archive" env \
+  HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$DEADMAN_CORRUPT_CAPTURE" \
+  python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
+    --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_CORRUPT_STATE" \
+    --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py"
+cp "$VALID_LAYOUT/archive/$SECOND_BATCH/database.dump" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
+chmod 0400 "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
+pass "ARM dead-man detects archive corruption independently of source state"
+mv "$DEADMAN_LAYOUT/archive/$SECOND_BATCH" "$TEST_DIR/deadman-withheld-batch"
+expect_failure "ARM dead-man detects that the source no longer produces a finalized archive" \
+  "${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
+    --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
+    --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py"
+[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "1" ]]
+expect_failure "ARM dead-man suppresses an identical repeated failure alert" \
+  "${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
+    --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
+    --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py"
+[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "1" ]]
+pass "ARM dead-man keeps repeated identical failure alerts deduplicated"
+mv "$TEST_DIR/deadman-withheld-batch" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH"
+"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
+  --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
+  --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null
+[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "2" ]]
+pass "ARM dead-man sends one recovery notification after finalized archives resume"
+
 DRIFT_BUILD_SHA="0123456789abcdef0123456789abcdef01234567"
 expect_failure "wrapper rejects an active runtime build that drifted from its installed contract" \
  "${wrapper_env[@]}" BUILD_SHA="$DRIFT_BUILD_SHA" VITE_BUILD_SHA="${DRIFT_BUILD_SHA:0:12}" \
@@ -60,6 +60,7 @@ validate_layout() {
  [[ "$DAILY_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "DAILY_KEEP must be positive"
  [[ "$WEEKLY_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "WEEKLY_KEEP must be positive"
  command -v "$PYTHON_BIN" >/dev/null 2>&1 || fail "python3 is required"
+  command -v flock >/dev/null 2>&1 || fail "flock is required"
 }

 validate_run_id() {
@@ -155,11 +156,44 @@ prune_named_directories() {
  done
 }

+publish_weekly_checkpoint() {
+  local archive_dir="$1"
+  local weekly_dir="$2"
+  local replace_existing="$3"
+  if [[ -e "$weekly_dir" && "$replace_existing" != "1" ]]; then
+    return 0
+  fi
+  local weekly_temp="${weekly_dir}.new.$$"
+  local weekly_rollback="${weekly_dir}.rollback.$$"
+  [[ ! -e "$weekly_temp" && ! -e "$weekly_rollback" ]] || return 1
+  if ! cp -al -- "$archive_dir" "$weekly_temp"; then
+    rm -rf -- "$weekly_temp"
+    return 1
+  fi
+  if [[ -e "$weekly_dir" ]]; then
+    [[ -d "$weekly_dir" && ! -L "$weekly_dir" ]] || return 1
+    mv -- "$weekly_dir" "$weekly_rollback" || return 1
+  fi
+  if ! mv -- "$weekly_temp" "$weekly_dir"; then
+    rm -rf -- "$weekly_temp"
+    if [[ -e "$weekly_rollback" ]]; then
+      mv -- "$weekly_rollback" "$weekly_dir"
+    fi
+    return 1
+  fi
+  if [[ -e "$weekly_rollback" ]]; then
+    rm -rf -- "$weekly_rollback"
+  fi
+}
+
 finalize_batch() {
  local run_id="$1"
  local expected_manifest_sha="$2"
  validate_run_id "$run_id"
  validate_layout
+  exec 9>"$BACKUP_ARCHIVE_DIR/.finalize.lock"
+  chmod 0600 "$BACKUP_ARCHIVE_DIR/.finalize.lock"
+  flock -n 9 || fail "another finalize operation is already running"

  local batch_name="hyb-oa-backup-$run_id"
  local partial_dir="$BACKUP_INBOX_DIR/${batch_name}.partial"
@@ -173,15 +207,32 @@ finalize_batch() {

  local run_day="${run_id#auto-}"
  run_day="${run_day%%T*}"
-  local existing_for_day
-  existing_for_day="$(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 -type d \
-    -name "hyb-oa-backup-auto-${run_day}T*Z" ! -name "$batch_name" -print -quit)"
-  [[ -z "$existing_for_day" ]] || fail "another automatic archive already exists for this UTC day"
+  local existing_for_day="" replacement_rollback=""
+  local -a existing_for_day_entries=()
+  mapfile -t existing_for_day_entries < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 -type d \
+    -name "hyb-oa-backup-auto-${run_day}T*Z" ! -name "$batch_name" -print)
+  [[ "${#existing_for_day_entries[@]}" -le 1 ]] || fail "daily archive invariant is already violated"
+  if [[ "${#existing_for_day_entries[@]}" -eq 1 ]]; then
+    existing_for_day="${existing_for_day_entries[0]}"
+  fi

  if [[ "$already_archived" == "0" ]]; then
    validate_batch "$partial_dir" "$run_id" "$expected_manifest_sha"
    chmod 0400 "$partial_dir/database.dump" "$partial_dir/storage.tar.gz" "$partial_dir/manifest.json"
-    mv -- "$partial_dir" "$archive_dir"
+    if [[ -n "$existing_for_day" ]]; then
+      [[ "$existing_for_day" =~ /hyb-oa-backup-auto-${run_day}T[0-9]{6}Z$ \
+        && -d "$existing_for_day" && ! -L "$existing_for_day" ]] \
+        || fail "existing daily archive is invalid"
+      replacement_rollback="$BACKUP_ARCHIVE_DIR/.daily-replacement-${run_day}.$$"
+      [[ ! -e "$replacement_rollback" ]] || fail "daily replacement rollback target already exists"
+      mv -- "$existing_for_day" "$replacement_rollback"
+    fi
+    if ! mv -- "$partial_dir" "$archive_dir"; then
+      if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then
+        mv -- "$replacement_rollback" "$existing_for_day"
+      fi
+      fail "daily archive publication failed"
+    fi
  fi

  local iso_week
@@ -192,10 +243,17 @@ day = datetime.strptime(sys.argv[1], "%Y%m%d").date()
 iso = day.isocalendar()
 print(f"week-{iso.year:04d}-W{iso.week:02d}")
 PYTHON
-)"
+  )"
  local weekly_dir="$BACKUP_WEEKLY_DIR/$iso_week"
-  if [[ ! -e "$weekly_dir" ]]; then
-    cp -al -- "$archive_dir" "$weekly_dir"
+  if ! publish_weekly_checkpoint "$archive_dir" "$weekly_dir" "$([[ -n "$replacement_rollback" ]] && echo 1 || echo 0)"; then
+    if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then
+      rm -rf -- "$archive_dir"
+      mv -- "$replacement_rollback" "$existing_for_day"
+    fi
+    fail "weekly checkpoint update failed"
+  fi
+  if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then
+    rm -rf -- "$replacement_rollback"
  fi

  prune_named_directories "$BACKUP_ARCHIVE_DIR" "hyb-oa-backup-auto-" "$DAILY_KEEP"
@@ -15,9 +15,10 @@ AUTOMATIC_BACKUP_STATE_DIR="${AUTOMATIC_BACKUP_STATE_DIR:-}"
 OFFHOST_TARGET="${OFFHOST_TARGET:-}"
 OFFHOST_SSH_KEY="${OFFHOST_SSH_KEY:-}"
 OFFHOST_KNOWN_HOSTS="${OFFHOST_KNOWN_HOSTS:-}"
+ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
+ALERT_HELPER="${ALERT_HELPER:-$ROOT_DIR/scripts/send-wecom-group-alert.py}"
 AUTOMATIC_BACKUP_KEEP="${AUTOMATIC_BACKUP_KEEP:-7}"
 AUTOMATIC_BACKUP_MAX_AGE_SECONDS="${AUTOMATIC_BACKUP_MAX_AGE_SECONDS:-86400}"
-ALERT_RECIPIENT_EMPLOYEE_NO="${ALERT_RECIPIENT_EMPLOYEE_NO:-HYD000006}"
 MODE="${1:-run}"

 CURRENT_PHASE="preflight"
@@ -79,7 +80,12 @@ validate_common() {
    || fail "frontend build SHA must match the full build prefix"
  [[ "$AUTOMATIC_BACKUP_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "backup retention must be positive"
  [[ "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" =~ ^[1-9][0-9]*$ ]] || fail "stale threshold must be positive"
-  [[ "$ALERT_RECIPIENT_EMPLOYEE_NO" =~ ^[A-Za-z0-9_-]{1,80}$ ]] || fail "alert recipient id is invalid"
+  validate_absolute_path ALERT_WEBHOOK_FILE "$ALERT_WEBHOOK_FILE"
+  validate_absolute_path ALERT_HELPER "$ALERT_HELPER"
+  [[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
+    && "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" ]] \
+    || fail "alert webhook file must exist with mode 0600"
+  [[ -f "$ALERT_HELPER" && ! -L "$ALERT_HELPER" ]] || fail "alert helper is missing"
  if [[ "$AUTOMATIC_BACKUP_DIR" == "$AUTOMATIC_BACKUP_STATE_DIR" \
    || "$AUTOMATIC_BACKUP_DIR" == "$AUTOMATIC_BACKUP_STATE_DIR"/* \
    || "$AUTOMATIC_BACKUP_STATE_DIR" == "$AUTOMATIC_BACKUP_DIR"/* ]]; then
@@ -218,10 +224,25 @@ PYTHON
 send_summary_alert() {
  local state_name="$1"
  [[ -n "$SUMMARY_FILE" && -f "$SUMMARY_FILE" ]] || return 1
-  [[ -n "$SERVER_CONTAINER" ]] || resolve_active_runtime
-  compose exec -T server sh -lc \
-    "cd /app/server && WARCH3_OPS_ALERT_SEND=1 npm run send:wecom-ops-alert -- --summary-file=- --recipient-employee-no=${ALERT_RECIPIENT_EMPLOYEE_NO} --send --state-file=/app/artifacts/qa-flow/${state_name} --format=json" \
-    <"$SUMMARY_FILE" >/dev/null
+  python3 "$ALERT_HELPER" \
+    --webhook-file "$ALERT_WEBHOOK_FILE" \
+    --summary-file "$SUMMARY_FILE" \
+    --state-file "$AUTOMATIC_BACKUP_STATE_DIR/$state_name" >/dev/null
+}
+
+capture_command() {
+  local phase="$1"
+  shift
+  local temporary="$AUTOMATIC_BACKUP_STATE_DIR/.${phase}.diagnostic.$$"
+  : >"$temporary"
+  chmod 0600 "$temporary"
+  if "$@" >"$temporary" 2>&1; then
+    rm -f -- "$temporary"
+    return 0
+  fi
+  mv -f -- "$temporary" "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log"
+  chmod 0600 "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log"
+  return 1
 }

 on_error() {
@@ -320,11 +341,12 @@ print(f"{unquote(value.username)}\t{database_name}")
  [[ "$source_image_digest" =~ ^sha256:[0-9a-f]{64}$ ]] || fail "active server image digest is invalid"

  CURRENT_PHASE="paired_backup"
-  if ! DATABASE_URL="$database_url" DB_TOOL_MODE=docker DB_CONTAINER="$DB_CONTAINER" \
+  if ! capture_command paired_backup env \
+    DATABASE_URL="$database_url" DB_TOOL_MODE=docker DB_CONTAINER="$DB_CONTAINER" \
    POSTGRES_DB="$postgres_db" POSTGRES_USER="$postgres_user" POSTGRES_PASSWORD="$postgres_password" \
    STORAGE_DIR_HOST="$STORAGE_HOST_DIR" BACKUP_DIR="$AUTOMATIC_BACKUP_DIR" \
    BACKUP_RUN_ID="$run_id" BUILD_SHA="$BUILD_SHA" SOURCE_IMAGE_DIGEST="$source_image_digest" \
-    "$ROOT_DIR/scripts/backup-all.sh" >/dev/null 2>&1; then
+    "$ROOT_DIR/scripts/backup-all.sh"; then
    fail "paired backup command failed"
  fi
  [[ -d "$batch_dir" && -f "$batch_dir/manifest.json" ]] || fail "complete local batch was not published"
@@ -343,14 +365,14 @@ print(f"{unquote(value.username)}\t{database_name}")
    "$OFFHOST_SSH_KEY" "$OFFHOST_KNOWN_HOSTS"

  CURRENT_PHASE="offhost_transfer"
-  if ! rsync -a --partial --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= -e "$rsync_shell" \
-    "$batch_dir/" "$OFFHOST_TARGET:${batch_name}.partial/" >/dev/null 2>&1; then
+  if ! capture_command offhost_transfer rsync -a --partial --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= -e "$rsync_shell" \
+    "$batch_dir/" "$OFFHOST_TARGET:${batch_name}.partial/"; then
    fail "off-host transfer failed"
  fi

  CURRENT_PHASE="offhost_finalize"
-  if ! ssh "${ssh_options[@]}" "$OFFHOST_TARGET" \
-    "hyb-backup-finalize $run_id $manifest_sha" >/dev/null 2>&1; then
+  if ! capture_command offhost_finalize ssh "${ssh_options[@]}" "$OFFHOST_TARGET" \
+    "hyb-backup-finalize $run_id $manifest_sha"; then
    fail "off-host finalize failed"
  fi

@@ -359,6 +381,7 @@ print(f"{unquote(value.username)}\t{database_name}")
  prune_local_backups "$batch_name"
  write_summary "PASS_AUTOMATIC_BACKUP_COMPLETE" 0 "offhost_verified"
  send_summary_alert "automatic-backup-alert-state.json" || true
+  rm -f -- "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log"
  trap - ERR
  echo "[automatic-backup][PASS] paired batch finalized off-host"
 }
@@ -369,9 +392,11 @@ check_stale() {
  CURRENT_PHASE="stale_check"
  resolve_active_runtime
  local success_file="$AUTOMATIC_BACKUP_STATE_DIR/last-success.json"
-  if ! python3 - "$success_file" "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" "$COMPOSE_PROJECT_NAME" "$BUILD_SHA" <<'PYTHON'
+  local stale_result
+  if ! stale_result="$(python3 - "$success_file" "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" "$COMPOSE_PROJECT_NAME" "$BUILD_SHA" <<'PYTHON'
 import json
 import os
+import re
 import sys
 from datetime import datetime, timezone

@@ -382,15 +407,29 @@ with open(file_path, "r", encoding="utf-8") as handle:
    payload = json.load(handle)
 if payload.get("schemaVersion") != 1 or payload.get("status") != "complete":
    raise SystemExit(1)
-if payload.get("composeProject") != expected_project or payload.get("buildSha") != expected_build:
+if payload.get("composeProject") != expected_project:
+    raise SystemExit(1)
+if not re.fullmatch(r"auto-[0-9]{8}T[0-9]{6}Z", payload.get("runId", "")):
+    raise SystemExit(1)
+if not re.fullmatch(r"[0-9a-f]{64}", payload.get("manifestSha256", "")):
+    raise SystemExit(1)
+if not re.fullmatch(r"[0-9a-fA-F]{40,64}", payload.get("buildSha", "")):
    raise SystemExit(1)
 completed = datetime.fromisoformat(payload["completedAt"].replace("Z", "+00:00"))
 age = (datetime.now(timezone.utc) - completed).total_seconds()
 if age < 0 or age > int(max_age):
    raise SystemExit(1)
+print("BUILD_MISMATCH" if payload.get("buildSha") != expected_build else "FRESH")
 PYTHON
-  then
-    fail "latest off-host success is missing, invalid, mismatched, or stale"
+  )"; then
+    fail "latest off-host success is missing, invalid, project-mismatched, or stale"
+  fi
+  if [[ "$stale_result" == "BUILD_MISMATCH" ]]; then
+    write_summary "PASS_WITH_FINDINGS" 1 "recovery_point_build_differs"
+    send_summary_alert "automatic-backup-alert-state.json" || true
+    trap - ERR
+    echo "[automatic-backup][PASS_WITH_FINDINGS] off-host recovery point is fresh but build differs"
+    return 0
  fi
  write_summary "PASS_AUTOMATIC_BACKUP_FRESH" 0 "fresh_offhost_recovery_point"
  send_summary_alert "automatic-backup-alert-state.json" || true
@@ -0,0 +1,188 @@
+#!/usr/bin/env python3
+"""Send a redacted, deduplicated OA operations alert to a WeCom group robot."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import stat
+import sys
+import tempfile
+import urllib.parse
+import urllib.request
+from pathlib import Path
+
+
+SAFE_TOKEN = re.compile(r"^[A-Za-z0-9_-]{1,160}$")
+ALLOWED_STATUSES = {
+    "FAIL_AUTOMATIC_BACKUP",
+    "FAIL_OFFHOST_DEADMAN",
+    "PASS_AUTOMATIC_BACKUP_COMPLETE",
+    "PASS_AUTOMATIC_BACKUP_FRESH",
+    "PASS_OFFHOST_DEADMAN_FRESH",
+    "PASS_WITH_FINDINGS",
+}
+
+
+def fail(message: str) -> "NoReturn":
+    print(f"[wecom-group-alert][ERROR] {message}", file=sys.stderr)
+    raise SystemExit(1)
+
+
+def validate_private_file(path: Path, label: str) -> None:
+    try:
+        info = path.lstat()
+    except FileNotFoundError:
+        fail(f"{label} is missing")
+    if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
+        fail(f"{label} must be a regular non-symbolic file")
+    if stat.S_IMODE(info.st_mode) != 0o600:
+        fail(f"{label} mode must be 0600")
+    if info.st_uid != os.geteuid():
+        fail(f"{label} must be owned by the executing account")
+
+
+def read_webhook(path: Path) -> str:
+    validate_private_file(path, "webhook file")
+    try:
+        value = path.read_text(encoding="utf-8").strip()
+    except OSError:
+        fail("webhook file cannot be read")
+    if "\n" in value or "\r" in value:
+        fail("webhook file must contain one line")
+    parsed = urllib.parse.urlparse(value)
+    query = urllib.parse.parse_qs(parsed.query, strict_parsing=True)
+    if (
+        parsed.scheme != "https"
+        or parsed.hostname != "qyapi.weixin.qq.com"
+        or parsed.netloc != "qyapi.weixin.qq.com"
+        or parsed.path != "/cgi-bin/webhook/send"
+        or set(query) != {"key"}
+        or len(query["key"]) != 1
+        or not re.fullmatch(r"[A-Za-z0-9-]{16,128}", query["key"][0])
+        or parsed.fragment
+    ):
+        fail("webhook file format is invalid")
+    return value
+
+
+def load_summary(path: Path) -> tuple[str, str, int]:
+    validate_private_file(path, "summary file")
+    try:
+        payload = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        fail("summary file is invalid")
+    status = payload.get("status")
+    findings = payload.get("findings") or []
+    finding_count = payload.get("findingCount")
+    if status not in ALLOWED_STATUSES or not isinstance(finding_count, int):
+        fail("summary status is invalid")
+    if finding_count != len(findings) or finding_count not in {0, 1}:
+        fail("summary finding count is invalid")
+    category = "healthy"
+    if findings:
+        category = findings[0].get("detail", "")
+        if not isinstance(category, str) or not SAFE_TOKEN.fullmatch(category):
+            fail("summary category is invalid")
+    return status, category, finding_count
+
+
+def load_previous(path: Path) -> dict[str, object]:
+    if not path.exists():
+        return {}
+    validate_private_file(path, "alert state file")
+    try:
+        value = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        fail("alert state file is invalid")
+    return value if isinstance(value, dict) else {}
+
+
+def atomic_write(path: Path, payload: dict[str, object]) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
+    os.chmod(path.parent, 0o700)
+    fd, temp_name = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
+    try:
+        os.fchmod(fd, 0o600)
+        with os.fdopen(fd, "w", encoding="utf-8") as handle:
+            json.dump(payload, handle, ensure_ascii=True, indent=2)
+            handle.write("\n")
+        os.replace(temp_name, path)
+    except BaseException:
+        try:
+            os.unlink(temp_name)
+        except FileNotFoundError:
+            pass
+        raise
+
+
+def should_send(previous: dict[str, object], status: str, category: str) -> bool:
+    previous_status = previous.get("status")
+    previous_category = previous.get("category")
+    if status.startswith("FAIL_") or status == "PASS_WITH_FINDINGS":
+        return previous_status != status or previous_category != category
+    return isinstance(previous_status, str) and (
+        previous_status.startswith("FAIL_") or previous_status == "PASS_WITH_FINDINGS"
+    )
+
+
+def safe_text(status: str, category: str) -> str:
+    if status.startswith("FAIL_"):
+        headline = "内部 OA 运维告警"
+    elif status == "PASS_WITH_FINDINGS":
+        headline = "内部 OA 运维提醒"
+    else:
+        headline = "内部 OA 运维恢复通知"
+    return f"{headline}\n状态：{status}\n类别：{category}\n请按恢复卡检查，不要在群内粘贴原始日志。"
+
+
+def send(webhook: str, text: str) -> None:
+    body = json.dumps({"msgtype": "text", "text": {"content": text}}).encode("utf-8")
+    capture = os.environ.get("HYB_WECOM_ALERT_TEST_CAPTURE")
+    if capture:
+        if os.environ.get("HYB_WECOM_ALERT_TEST_MODE") != "1":
+            fail("test capture is disabled")
+        capture_path = Path(capture)
+        with capture_path.open("ab") as handle:
+            handle.write(body + b"\n")
+        os.chmod(capture_path, 0o600)
+        return
+    request = urllib.request.Request(
+        webhook,
+        data=body,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    try:
+        with urllib.request.urlopen(request, timeout=10) as response:
+            result = json.loads(response.read().decode("utf-8"))
+    except Exception:
+        fail("group robot request failed")
+    if result.get("errcode") != 0:
+        fail("group robot rejected the alert")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--webhook-file", required=True, type=Path)
+    parser.add_argument("--summary-file", required=True, type=Path)
+    parser.add_argument("--state-file", required=True, type=Path)
+    args = parser.parse_args()
+
+    try:
+        status, category, _ = load_summary(args.summary_file)
+        previous = load_previous(args.state_file)
+        send_now = should_send(previous, status, category)
+        if send_now:
+            webhook = read_webhook(args.webhook_file)
+            send(webhook, safe_text(status, category))
+        atomic_write(args.state_file, {"schemaVersion": 1, "status": status, "category": category})
+    except OSError:
+        fail("local alert file operation failed")
+    print(f"[wecom-group-alert][PASS] evaluated sent={str(send_now).lower()}")
+
+
+if __name__ == "__main__":
+    main()