完善异机备份告警与同日恢复点取代（07-27-automatic-offhost-backup）
这个提交包含在：
@@ -927,13 +927,18 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
|
||||
- Production runner: `scripts/run-automatic-backup.sh` with `run`,
|
||||
`check-stale` and `remind-drill` modes.
|
||||
- ARM forced receiver: `scripts/receive-offhost-backup.sh`.
|
||||
- Shared host alert helper: `scripts/send-wecom-group-alert.py` with explicit
|
||||
`--webhook-file`, `--summary-file` and `--state-file` paths.
|
||||
- ARM watchdog: `scripts/check-offhost-backup-deadman.py` with archive root,
|
||||
state root, alert helper, Webhook file and a default `93600` second threshold.
|
||||
- Installers: `install-automatic-backup-timer.sh` and
|
||||
`install-offhost-backup-receiver.sh`.
|
||||
- Local contract: `bash scripts/qa/run-automatic-backup-contract.sh`.
|
||||
- Production required environment: `REMOTE_DIR`, `REMOTE_ENV_FILE`,
|
||||
`COMPOSE_PROJECT_NAME`, `POSTGRES_DATA_DIR`, `STORAGE_HOST_DIR`, `BUILD_SHA`,
|
||||
`VITE_BUILD_SHA`, `AUTOMATIC_BACKUP_DIR`, `AUTOMATIC_BACKUP_STATE_DIR`,
|
||||
`OFFHOST_TARGET`, `OFFHOST_SSH_KEY` and `OFFHOST_KNOWN_HOSTS`.
|
||||
`OFFHOST_TARGET`, `OFFHOST_SSH_KEY`, `OFFHOST_KNOWN_HOSTS`,
|
||||
`ALERT_WEBHOOK_FILE` and `ALERT_HELPER`.
|
||||
- Receiver config: independent absolute `BACKUP_INBOX_DIR`,
|
||||
`BACKUP_ARCHIVE_DIR`, `BACKUP_WEEKLY_DIR`, `RRSYNC_BIN`, `PYTHON_BIN`,
|
||||
`DAILY_KEEP` and `WEEKLY_KEEP`.
|
||||
@@ -952,14 +957,32 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
|
||||
it rejects shell, sender/read and destructive rsync options.
|
||||
- ARM revalidates manifest identity, exact regular-file set, sizes and SHA-256,
|
||||
then atomically moves the batch to archive. Archive and weekly directories are
|
||||
outside the rsync root; an archived name is never overwritten.
|
||||
outside the rsync root. A different run ID for the same UTC day may replace
|
||||
that day's archive only after full candidate validation: rename the old daily
|
||||
point to a same-filesystem rollback name, publish the candidate, rebuild that
|
||||
ISO week's hard-link checkpoint, then remove the rollback. Any publication or
|
||||
weekly-checkpoint failure restores the prior daily point.
|
||||
- Write production `last-success.json` and prune source copies only after ARM
|
||||
finalize succeeds. Keep seven automatic source/ARM daily points and four ARM
|
||||
weekly checkpoints; unrelated manual batches are outside automatic retention.
|
||||
- Daily, freshness and quarterly reminder timers are separate. Alert summaries
|
||||
contain categories only. AUTO_BACKUP sends the first failure, worsening/new
|
||||
findings and the first quarterly reminder, while suppressing an identical
|
||||
repeated failure or reminder.
|
||||
contain categories only. A shared Python helper reads the complete WeCom group
|
||||
robot URL only from a regular, executing-account-owned `0600` file and never
|
||||
prints it. It sends the first failure, worsening/new findings, one recovery
|
||||
notice and the first quarterly reminder, while suppressing an identical
|
||||
repeated state. It does not call the OA server container or resolve an employee.
|
||||
- ARM runs its own 26-hour dead-man as root. It verifies the newest archive's
|
||||
exact file set, manifest identity, sizes, checksums and timestamp, so a stopped
|
||||
production host/server can still cause a group alert. Missing, corrupt, stale
|
||||
or future-dated archives fail; recovery sends once.
|
||||
- Production command diagnostics for paired backup, rsync and finalize stay only
|
||||
in the state directory as `last-error.log` mode `0600`. The summary and group
|
||||
message contain only the safe phase category. A later successful off-host run
|
||||
removes the retained diagnostic.
|
||||
- The production freshness check hard-fails missing, malformed, future, stale or
|
||||
Compose-project-mismatched state. A fresh recovery point with a different
|
||||
build SHA is `PASS_WITH_FINDINGS`, because deploy-time code identity and
|
||||
recoverable data freshness are separate facts.
|
||||
- Installer `install` is disabled-by-default; `enable`, real transfer, real
|
||||
WeCom alert and all server/account/key changes remain `EXTERNAL_GATED`.
|
||||
|
||||
@@ -967,24 +990,32 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
|
||||
|
||||
- Source mount/build/project drift, lock conflict or incomplete backup -> fail
|
||||
before success state and source retention.
|
||||
- Transfer/finalize failure, wrong/extra/link artifact, bad hash or second batch
|
||||
for one UTC day -> leave no off-host success state and preserve source batch.
|
||||
- Missing, stale or project/build-mismatched success record -> watchdog finding.
|
||||
- Transfer/finalize failure, wrong/extra/link artifact or bad hash -> leave no
|
||||
off-host success state, preserve the source batch and retain a root-only local
|
||||
diagnostic. A corrupt same-day candidate leaves the previous archive intact.
|
||||
- Valid same-day retry -> replace only that UTC day's point and rebuild its ISO
|
||||
weekly checkpoint; concurrent finalize -> `flock` rejection.
|
||||
- Missing, corrupt, stale, future or project-mismatched success record -> hard
|
||||
watchdog failure. Fresh build mismatch -> one finding without a failing exit.
|
||||
- Missing/non-regular/symlink/non-`0600` Webhook file -> fail closed without
|
||||
printing the file value. Identical alert state -> update/evaluate without send.
|
||||
- Ordinary shell, rsync sender/read/delete or archive overwrite -> receiver
|
||||
rejection without changing archive.
|
||||
- Host/network/WeCom all unavailable at once -> self-hosted alert cannot prove
|
||||
its own outage; this is a documented physical boundary, not a green state.
|
||||
- ARM host/network/WeCom all unavailable at once -> the group robot path cannot
|
||||
prove its own outage; this physical boundary is not a green state.
|
||||
|
||||
### 5. Good/Base/Bad Cases
|
||||
|
||||
- Good: one verified active source creates a complete batch, write-only rsync
|
||||
resumes inside inbox, ARM finalize verifies and archives it, then production
|
||||
writes success and prunes old automatic source points.
|
||||
writes success and prunes old automatic source points. If production later
|
||||
stops, ARM independently raises one dead-man alert and one recovery notice.
|
||||
- Base: local implementation and synthetic contracts pass while both server
|
||||
installers remain disabled and explicitly externally gated.
|
||||
- Bad: give production a normal ARM login key, put archive below inbox, accept
|
||||
a new host fingerprint automatically, or prune local recovery points after
|
||||
rsync but before target-side finalize.
|
||||
- Bad: send through `compose exec server`, hard-code a personal employee number,
|
||||
persist a robot URL in `.env`/unit/repository, give production a normal ARM
|
||||
login key, put archive below inbox, accept a new host fingerprint automatically,
|
||||
or prune local points before target-side finalize.
|
||||
|
||||
### 6. Tests Required
|
||||
|
||||
@@ -993,28 +1024,37 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
|
||||
- Docker mocks must return the real `{{json .Mounts}}` shape consumed by the
|
||||
runner. A simplified tab-delimited mock is not valid evidence for real mount
|
||||
parsing.
|
||||
- Focused WeCom alert tests for stdin path redaction, preserved AUTO_BACKUP
|
||||
issue identity, first/repeated failure and quarterly reminder behavior.
|
||||
- Environment contract assertions for all four scripts, three timers, explicit
|
||||
project/data/build/SSH binding and disabled-by-default installation.
|
||||
- Real ARM setup, first paired batch, failure alert and quarterly shadow restore
|
||||
are separately approved operational evidence.
|
||||
- Contract assertions for same-day replacement, corrupt-candidate preservation,
|
||||
weekly hard-link rebuild, source diagnostic mode, build mismatch finding,
|
||||
Webhook mode/symlink rejection and group alert behavior without a server
|
||||
container. Test transport must be explicitly gated and local-only.
|
||||
- Environment contract assertions for both helpers, both installers, production
|
||||
timers, ARM dead-man timer, explicit project/data/build/SSH/Webhook-file
|
||||
binding, disabled-by-default installation and absence of a literal complete
|
||||
robot URL in tracked or untracked repository files.
|
||||
- Real ARM setup, first paired batch, deliberate source-timer stop or transfer
|
||||
failure, dead-man first/repeated/recovery behavior and quarterly shadow restore
|
||||
are separately approved operational evidence. A direct robot send test alone
|
||||
is not dead-man evidence.
|
||||
|
||||
### 7. Wrong vs Correct
|
||||
|
||||
#### Wrong
|
||||
|
||||
```bash
|
||||
compose exec server send-alert --recipient=<personal-employee-number>
|
||||
rsync -a <batch>/ backup-host:<archive>/
|
||||
rm -rf <old-local-batches>
|
||||
```
|
||||
|
||||
#### Correct
|
||||
|
||||
```bash
|
||||
# Install only with explicit active identity; default remains disabled.
|
||||
<explicit-project-data-build-and-ssh-env> \
|
||||
# Install only with explicit active identity and a root-only Webhook file path;
|
||||
# default remains disabled.
|
||||
<explicit-project-data-build-ssh-and-webhook-file-env> \
|
||||
bash deploy/trialrun/install-automatic-backup-timer.sh install
|
||||
# ARM separately verifies archive freshness even when production is unavailable.
|
||||
python3 scripts/check-offhost-backup-deadman.py <explicit-safe-path-arguments>
|
||||
# The runner writes success and prunes only after restricted target finalize.
|
||||
bash scripts/run-automatic-backup.sh run
|
||||
```
|
||||
|
||||
@@ -19,7 +19,12 @@ production 02:30
|
||||
-> prune local daily copies
|
||||
|
||||
production 07:10 watchdog
|
||||
-> last off-host success <= 24h ? pass : alert
|
||||
-> recovery point missing/corrupt/stale/project drift ? fail + group alert
|
||||
-> build SHA differs ? one finding + group alert
|
||||
|
||||
ARM independent watchdog
|
||||
-> newest finalized archive <= 26h ? pass : direct group alert
|
||||
-> source host/server/container not required
|
||||
|
||||
quarterly reminder
|
||||
-> one redacted WeCom reminder per calendar quarter
|
||||
@@ -39,8 +44,8 @@ quarterly reminder
|
||||
- ARM 一次性安装 rsync，创建密码锁定的专用账号、独立 inbox/archive/weekly 目录和一把专用公钥。该公钥的 `authorized_keys` 强制执行仓库提供的 receiver，并禁用 PTY、端口转发、agent 转发和 X11 转发。
|
||||
- receiver 只接受两类 `SSH_ORIGINAL_COMMAND`：rsync server 接收模式，或 `hyb-backup-finalize <run-id> <manifest-sha256>`。其他命令全部拒绝。
|
||||
- rsync 通过 `rrsync -wo` 只能进入 inbox；archive 和 weekly 位于 rrsync 根之外。即使生产端密钥泄露，也不能用 rsync 读取或删除已归档批次。
|
||||
- finalize 只接受 `auto-` 安全批次号和完整 manifest SHA。它要求 inbox 中恰有三个普通文件、无链接，manifest 的 runId/status/schema、相对文件名、大小和 SHA-256 全部匹配；已存在 archive 目标时拒绝覆盖。
|
||||
- 校验通过后在同一文件系统内原子移动到 archive。每个自然日最多接受一个正式自动批次进入保留计算，避免失陷源端通过短时间伪造大量批次提前挤掉历史。
|
||||
- finalize 只接受 `auto-` 安全批次号和完整 manifest SHA。它要求 inbox 中恰有三个普通文件、无链接，manifest 的 runId/status/schema、相对文件名、大小和 SHA-256 全部匹配；同 run_id 重复 finalize 继续保持幂等。
|
||||
- 同 UTC 日已有不同 run_id 时，先完整校验并冻结新批次权限，再把旧日归档 rename 到同文件系统临时回滚名、新批次 rename 到正式归档名；新批次发布失败立即把旧归档 rename 回原名。新批次发布成功后才删除旧回滚名，并重建该 ISO 周硬链接检查点，使每日一个恢复点和周检查点都指向当前有效批次。
|
||||
- archive 保留最近 7 个日批次；每个 ISO 周首次成功批次用只读硬链接快照形成周检查点，保留最近 4 周。清理不得删除最后一个 archive 或当前 finalize 批次。
|
||||
- 旧手工/审计批次不进入自动前缀和自动轮转范围。
|
||||
|
||||
@@ -52,19 +57,22 @@ quarterly reminder
|
||||
|
||||
## 告警与静默失败检测
|
||||
|
||||
- wrapper 无论成功或失败都生成字段白名单 summary；失败 trap 尝试通过活动 server 容器调用现有企微运维告警 CLI，并使用备份专属 state file，避免与 WARCH3 降噪状态互相覆盖。
|
||||
- 现有告警 CLI 小范围泛化为接受 stdin summary、保留 summary issueId，并为备份失败/过期提供正确建议；发送仍受既有 `WARCH3_OPS_ALERT_SEND=1` 双门控制。
|
||||
- 独立 watchdog timer 在每日巡检前读取宿主 `last-success.json`。文件缺失、格式错误、构建/项目不匹配或成功时间超过 24 小时均产生失败 summary 并尝试告警。
|
||||
- 若 Docker、整机网络或企微本身同时不可用，主机无法自证并外发，这是自托管告警的物理边界；watchdog 能覆盖 timer 未运行和短暂失败，但不替代外部主机存活监控。
|
||||
- 新增一个小型宿主告警 helper，使用 Python 标准库从 root-only Webhook 文件读取完整 URL，并向企微运维群机器人发送固定白名单文本；不复用 OA server 容器、员工表或个人工号。
|
||||
- 生产 wrapper 无论成功或失败都生成字段白名单 summary；即时失败和源端 watchdog 直接调用宿主 helper。Webhook 文件缺失、权限不是 `0600`、不是普通文件或内容格式不合法时 fail closed，并在本机诊断中留类别，不打印 URL。
|
||||
- ARM receiver 安装器同时安装独立 dead-man service/timer。watchdog 读取 archive 中最新有效自动批次的完成时间，超过 26 小时、没有批次或 manifest 无法验证时直接调用同一 helper；状态文件实现首次失败、重复降噪和恢复通知。
|
||||
- 生产 stale watchdog 仍验证 schema/status/project/时间。build SHA 不同输出 `PASS_WITH_FINDINGS`，不返回失败；恢复点缺失、损坏、未来时间、超过阈值或 project 不同继续返回失败。
|
||||
- 生产外部命令的 stdout/stderr 只写状态目录下的临时诊断，失败时原子保留为 `last-error.log` 并设为 `0600`，成功后清理本次临时诊断。summary 和企微文本不得复制诊断正文。
|
||||
- Webhook secret 同时存在于两台主机的 root-only 文件，仓库只保存“文件路径”合同；`.env.example` 和 systemd `Environment=` 都不得出现 URL/key。
|
||||
- 季度提醒以年份+季度作为 finding key，确保每季度一次且同季度不重复；只提醒人工影子恢复，不执行恢复。
|
||||
|
||||
## 安装与回滚
|
||||
|
||||
1. 本地先实现合成测试、受限 receiver、告警适配和两个安装器，不接触生产/ARM。
|
||||
2. 获批后先在 ARM 安装 rsync、受限账号/receiver/公钥和目标目录；使用无业务数据的合成批次验证拒绝 shell、拒绝 archive 覆盖和 finalize。
|
||||
3. 再在生产安装专用私钥/known_hosts、每日备份 service/timer、stale watchdog 和季度提醒；首次安装先保持 timer disabled。
|
||||
4. 手工启动一次获批自动备份，核对源/ARM 哈希、状态和无敏感输出；再 enable timers。
|
||||
5. 用不包含业务正文的可控失败验证企微告警后恢复链路，最终确认下次执行时间。
|
||||
2. 获批后先由用户在企微创建运维群机器人，并分别把同一 Webhook 写入两台主机 root-only 文件；不向 Codex、仓库或普通 `.env` 提供其内容。
|
||||
3. 在 ARM 安装 rsync、受限账号/receiver/公钥、dead-man timer 和目标目录；使用无业务数据的合成批次验证拒绝 shell、同日安全取代和 dead-man 状态逻辑。
|
||||
4. 再在生产安装专用私钥/known_hosts、每日备份 service/timer、stale watchdog 和季度提醒；首次安装先保持 timer disabled。
|
||||
5. 手工启动一次获批自动备份，核对源/ARM 哈希、状态和无敏感输出；再 enable timers。
|
||||
6. 真正停止一次源端 timer 或故意让一次传输失败，等待/模拟超过 26 小时并确认 ARM 群告警；恢复链路后验证恢复通知和新批次成功。
|
||||
|
||||
回滚只 disable/stop 新 timer、移除 unit 和专用密钥授权；保留已生成的完整备份、状态报告和 ARM archive，不删除历史数据。旧手工 `backup-all.sh` 入口保持兼容。
|
||||
|
||||
@@ -74,3 +82,4 @@ quarterly reminder
|
||||
- 不给生产普通 ARM shell，而用 inbox + 强制 receiver + archive 隔离，抵消推送方案最主要的跨机删除风险。
|
||||
- 使用 rsync 而不是新备份服务；当前压缩批次规模较小，收益主要是断点续传和成熟 SSH 传输，不追求块级去重。
|
||||
- 不自动恢复。季度人工影子恢复才是可恢复性证据，日 timer 只负责生成恢复点。
|
||||
- 运维加固在 dead-man、真实失败链路、timer、恢复卡第二人走查完成后硬收口；不继续新增平台化备份或告警能力，后续回到产品 backlog。
|
||||
|
||||
@@ -9,9 +9,16 @@
|
||||
- [x] 增加合成契约测试：来源漂移、锁冲突、不完整/损坏/重复批次、续传 partial、拒绝普通 shell、保留轮转、stale 和季度降噪。
|
||||
- [x] 更新部署环境契约、runbook、checklist、管理员手册的最小操作入口和 `.trellis/spec/hyb-oa/deployment-trialrun.md`；不新增平行长文档。
|
||||
- [x] 生成一页脱敏恢复卡内容，优先放在现有 runbook 的首部紧急段；由用户复制到公司知识库并安排第二人走查。
|
||||
- [ ] 完成本地全量门禁并提交；生产、ARM 安装另行请求批准。
|
||||
- [x] 完成本地全量门禁并提交；生产、ARM 安装另行请求批准。
|
||||
- [ ] 获批后先安装 ARM 接收端并用合成批次验证，再安装生产 units，手工运行首次真实自动批次，最后 enable timers。
|
||||
- [ ] 获批执行一次可控传输失败告警并恢复正常链路；不读取附件正文、不执行业务或企微其他写入。
|
||||
- [x] 删除个人工号默认值，新增生产/ARM 共用的 root-only 企微群机器人 helper；两端 systemd 只绑定 Webhook 文件路径。
|
||||
- [x] 增加 ARM 26 小时 dead-man service/timer，覆盖缺批次、超龄、损坏、重复降噪和恢复通知。
|
||||
- [x] 将 buildSha 不同降级为 finding；缺失/损坏/超龄/project 不同继续硬失败，并支持部署后立即运行一次备份。
|
||||
- [x] 为 backup/rsync/finalize 保存 `0600` 最近失败诊断，summary 与群消息只保留脱敏类别。
|
||||
- [x] 把同 UTC 日不同 run_id 从拒绝改为“完整校验后原子取代”，覆盖失败回滚和周检查点重建。
|
||||
- [x] 增加 Webhook 不进入 `.env.example`/仓库配置/报告的否定断言，并在 runbook 记录调整 `BACKUP_RUN_AT` 后需人工确认次日窗口。
|
||||
- [x] 本地门禁通过并完成只读复核后提交；不推送、不部署、不配置真实 Webhook。
|
||||
|
||||
## 预计文件边界
|
||||
|
||||
@@ -22,6 +29,7 @@
|
||||
- 新增 `scripts/qa/run-automatic-backup-contract.sh`：纯合成正负向测试。
|
||||
- 修改 `server/src/scripts/send-wecom-ops-alert.js` 及其单测，仅做通用脱敏 summary 兼容。
|
||||
- 修改 `deploy/trialrun/validate-env-contract.js`、现有 runbook/checklist/管理员手册和部署规约。
|
||||
- 新增一个不含 Secret 的宿主告警 helper；只读取安装时传入的 root-only Webhook 文件。
|
||||
|
||||
## 验证命令
|
||||
|
||||
@@ -37,28 +45,31 @@
|
||||
- `node scripts/qa/audit-docs.mjs --check-registry`
|
||||
- `node scripts/qa/check-process-contract.mjs --baseline --json`
|
||||
- `git diff --check`
|
||||
- 合成测试必须模拟源端不再产生 finalize，并由 ARM dead-man 得到失败；不能只直接调用“发送消息”路径。
|
||||
|
||||
## 外部门禁与回滚点
|
||||
|
||||
- 安装 ARM rsync/账号/authorized_keys/目录属于远端配置写入，须单独批准；不得复用普通登录账号或本地工作站私钥。
|
||||
- 安装生产 SSH key/config/systemd units、首次真实备份、真实异机传输和企微失败告警均须单独批准。
|
||||
- 创建企微运维群机器人及把 Webhook 写入两台主机 root-only 文件由用户手工完成；Codex 不读取、不回显、不记录其内容。
|
||||
- 首次真实执行前记录现有手工批次数量、磁盘余量、活动容器/mount/build 和现有 timer；只输出脱敏计数/布尔值。
|
||||
- 任何目标身份、主机指纹、活动 mount、manifest 或哈希不匹配立即停止，不猜路径、不接受新指纹、不清理源端批次。
|
||||
- 回滚新 timer/receiver 时只撤销调度和专用授权；ARM archive、源端完整批次和原 WARCH3 timer 保留。
|
||||
|
||||
## 本地实施与验证证据
|
||||
|
||||
- 已实现生产 wrapper、ARM forced receiver、两端安装器和 3 组 systemd timer；安装器默认只安装 disabled units。
|
||||
- `run-automatic-backup-contract.sh` 使用临时目录与 mock Docker/SSH/rsync 完成 28 项正负向合同，不连接真实主机。
|
||||
- 已实现生产 wrapper、ARM forced receiver、独立群机器人 helper、ARM dead-man 和两端安装器；生产 3 组 timer 与 ARM 1 组 timer 均默认只安装为 disabled。
|
||||
- `run-automatic-backup-contract.sh` 使用临时目录与 mock Docker/SSH/rsync 完成 46 项正负向合同，不连接真实主机；覆盖同日取代与回滚、坏候选保留、周检查点重建、Webhook 权限/链接/格式负例、build finding、状态损坏、源端停止、重复降噪和恢复通知。
|
||||
- 企微告警聚焦单测 15 项通过；stdin 路径被替换为安全类别，AUTO_BACKUP 首次失败/季度提醒发送且相同重复状态降噪。
|
||||
- `run-backup-restore-drill.sh` 本地合成 DB/附件往返通过，本次合成 RTO 为 8 秒；该结果不是生产 RTO 或 ARM 恢复证据。
|
||||
- `scripts/check-minimal.sh` 通过：server 162 个测试文件、1004 项测试全部通过，client typecheck/build 通过。
|
||||
- 环境合同、Trellis task/governance、敏感扫描、文档注册、流程合同和 `git diff --check` 均通过；文档注册保留既有 22 个未注册文件 warning，无本任务新增文档。
|
||||
- ARM 安装、生产 units、真实首批、timer enable、真实企微失败告警和第二人季度走查仍为外部门禁，未执行、不得写成已完成。
|
||||
- 最终只读复核修正了三个本地缺陷：生产安装器残留个人工号校验、ARM 重装可能改变 dead-man 状态文件属主、stale 状态未校验 run id/manifest SHA；修正后 46 项合同与 `scripts/check-minimal.sh` 全量复跑通过。
|
||||
- 更新后的 ARM dead-man/helper、生产 helper/runner、真实有效首批、四组 timer enable、真实企微失败链路和第二人季度走查仍为外部门禁，未执行、不得写成已完成。
|
||||
|
||||
### 首次远端执行增量
|
||||
|
||||
- ARM 已安装 rsync、密码锁定专用账号和 forced-command receiver；从生产验证普通 shell、sender/read、delete 均拒绝，写通道 dry-run 通过且未生成归档。
|
||||
- 生产三组 units 已安装为 disabled/inactive，并绑定活动 Compose project、PGDATA、附件挂载、构建标识和固定 SSH 身份。
|
||||
- 首次真实备份在 `runtime_identity` 阶段 fail-closed，未生成源端或 ARM 批次；真实 Docker `println` 输出的参数间空格导致制表符拆分后路径不精确相等。
|
||||
- runner 已改为解析 `docker inspect` mount JSON 并要求唯一匹配；合成 mock 同步改为真实 JSON 形状。重新执行真实首批和 timer enable 尚待验证。
|
||||
- 首次真实备份曾在 `runtime_identity` 阶段因 Docker `println` 参数间空格 fail-closed；runner 改用 `docker inspect` mount JSON 后，后续真实源端成对批次已生成并传输。
|
||||
- ARM finalize 随后因同 UTC 日已有不同 run id 而拒绝，因此尚无可作为自动备份闭环证据的新批次；本轮已在本地实现“完整校验后同日取代”，更新后的远端脚本、首批和 timer enable 仍待另行批准。
|
||||
|
||||
@@ -12,6 +12,8 @@
|
||||
- 两台主机之间当前没有可直接复用的免密 SSH 别名或密钥链路，不能把本地工作站的 SSH 配置误当作服务器间自动化能力。
|
||||
- 当前手工正确批次和 ARM 影子恢复已证明格式可用，但不能满足 RPO 24 小时或自动失败发现。
|
||||
- 已确认采用“生产主动推送到 ARM 的专用受限接收身份”：生产保留即时传输结果，ARM 的已归档批次不暴露给 rsync 写入根。
|
||||
- 生产切换已经完成并应用 61 项 migration；本地当前为 62 项，`2526052` 后续部署只新增 1 项 migration。自动备份三组 units 已安装但仍未形成可靠启用闭环。
|
||||
- 首次真实自动批次已证明源端成对备份可生成，但 ARM 因“同一 UTC 日仅允许一个不同 run_id”拒绝重试批次；现有即时告警还依赖活动 server 容器，不能覆盖容器或整机故障。
|
||||
|
||||
## Scope
|
||||
|
||||
@@ -28,7 +30,11 @@
|
||||
- 完整批次通过固定主机指纹和专用受限密钥自动推送到 ARM；该身份无普通 shell，只能把文件写入 inbox 或请求校验/归档，不能通过 rsync 读取或删除 archive。
|
||||
- 传输端和目标端都要校验 manifest、文件大小和 SHA-256；同名已归档目标不得覆盖。中断传输可以在 inbox 内续传，但只有完整校验后才能原子进入 archive。
|
||||
- 生产保留最近 7 个成功日批次；ARM 至少保留最近 7 个日批次和 4 个周检查点。清理只能发生在本次异机成功确认之后，且不得删除最后一个已知成功批次。
|
||||
- 备份或异机校验失败必须生成脱敏状态报告，并复用现有企微运维告警能力；独立 stale watchdog 应在现有 07:30 巡检前检查最近一次异机成功时间，防止备份 timer 未运行或即时告警自身失败造成静默缺口。
|
||||
- 生产即时告警和 ARM dead-man 告警统一使用独立企微运维群机器人；Webhook 只从 root-only 文件读取，不依赖 OA server、数据库、员工绑定或默认个人工号，也不得进入仓库、`.env.example`、Trellis、日志或聊天。
|
||||
- ARM 安装独立 26 小时 dead-man timer，直接检查最近归档完成时间；生产整机或容器不可用、timer 未运行、传输未 finalize 时仍能从另一台主机发出告警。
|
||||
- 生产 watchdog 的“恢复点缺失/损坏/超龄/Compose project 不匹配”继续硬失败；仅 `buildSha` 早于当前部署时降级为一次 finding，不得误报为没有恢复点。部署后仍推荐立即触发一次获批备份。
|
||||
- backup、rsync 和 finalize 的失败输出必须写入状态目录下权限 `0600` 的最近失败诊断；企微告警仍只包含阶段、状态、finding 数和脱敏建议，不附原始输出。
|
||||
- 同一 UTC 日已有不同自动归档时，新批次必须先完整通过 manifest、文件集、大小和 SHA-256 校验，再以可回滚的同文件系统 rename 顺序原子取代旧日批次；任一替换步骤失败必须恢复旧批次或保留可诊断状态。最终仍只保留每日一个恢复点。
|
||||
- 告警只包含阶段、状态、finding 数和脱敏建议，不包含数据库凭证、完整目录、SSH 信息、真实业务数据、附件名或原始命令输出。
|
||||
- 季度恢复保持人工触发，在 ARM 隔离目标执行；季度 timer 只发送一次脱敏提醒，不得自动运行恢复，更不得触碰生产库。
|
||||
- 形成一页脱敏恢复卡，说明主机故障后的负责人、备份位置类别、恢复顺序和验收项，供另一名人员存入公司知识库。
|
||||
@@ -38,7 +44,11 @@
|
||||
- [x] 本地合成测试证明每日 wrapper 只接受唯一活动来源、互斥锁有效，并只发布完整备份批次。
|
||||
- [x] 传输测试证明不完整批次、哈希不一致、重复批次和不可达 ARM 均失败，且不产生异机成功状态。
|
||||
- [x] 保留测试覆盖 7 个日批次、4 个周检查点、最后成功批次保护和仅在异机确认后清理。
|
||||
- [x] 告警测试证明首次失败和状态恶化会告警，重复同一失败降噪；07:30 stale 检查能发现超过 24 小时没有异机成功记录。
|
||||
- [x] 告警测试证明首次失败和状态恶化会告警，重复同一失败降噪；07:10 stale 检查能发现超过 24 小时没有异机成功记录。
|
||||
- [x] 本地合成测试证明同日不同 run_id 的完整新批次可取代旧批次，损坏新批次不会影响旧归档，且周检查点与日归档不产生悬空或错误指向。
|
||||
- [x] 本地合成测试证明 buildSha 不同仅产生 finding，恢复点缺失/损坏/超龄/project 不同仍失败；失败诊断为 `0600` 且不进入告警摘要。
|
||||
- [x] 环境契约证明 Webhook 相关键和值不出现在 `.env.example`、仓库配置或报告；两端安装器只接受 root-only Webhook 文件路径。
|
||||
- [x] ARM 26 小时 dead-man 合成验证覆盖“源端停止/不产生新 finalize”而非仅验证机器人能发送，并覆盖首次失败、重复降噪与恢复通知。
|
||||
- [ ] systemd 安装器显式绑定 Compose project、活动数据路径、构建标识、备份根和受限远端身份；安装后 timer enabled/active，但部署验证不手工执行真实告警。
|
||||
- [ ] 经单独生产授权后完成一次自动日备份：源端完整、ARM 目标端哈希一致、状态报告脱敏，且未读取附件正文或修改业务数据。
|
||||
- [ ] 模拟传输失败能在当天产生一条脱敏企微告警；测试后恢复正常链路并完成成功批次。
|
||||
@@ -49,12 +59,15 @@
|
||||
- 不引入备份平台、对象存储、MinIO、云厂商 SDK、Kubernetes 或持续复制。
|
||||
- 不自动执行生产恢复、数据库 reset、迁移改写或附件回写。
|
||||
- 不把真实业务正文、附件清单、凭证、主机地址、完整目录或 SSH 私钥写入仓库、报告或聊天。
|
||||
- 不把企微群机器人 Webhook URL、key 或配置文件正文写入仓库、`.env.example`、Trellis、报告、日志或聊天。
|
||||
- 不在本任务继续企微本人授权、员工角色、付款、报销或其他产品功能。
|
||||
- 不因自动备份完成而取消季度人工影子恢复。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Shell 语法与合成文件系统测试，覆盖锁、来源解析、传输失败、目标重复、哈希和保留轮转。
|
||||
- 合成 receiver 测试覆盖同日完整批次取代、损坏候选保留旧批次、替换失败恢复以及周检查点一致性。
|
||||
- 合成双主机告警测试覆盖生产无 server 容器、ARM 26 小时无新归档、重复降噪、恢复通知和 root-only Webhook 文件门禁。
|
||||
- 复用 `scripts/qa/run-backup-restore-drill.sh` 验证成对备份合同未回退。
|
||||
- 聚焦测试现有企微运维告警的备份状态与 stale finding 适配。
|
||||
- `node deploy/trialrun/validate-env-contract.js`。
|
||||
|
||||
@@ -31,7 +31,7 @@
|
||||
- [ ] 正式 deploy 使用绝对数据路径；既有实例使用 `DEPLOY_DATA_MODE=existing`，PGDATA 已存在且含 `PG_VERSION`，附件目录已存在
|
||||
- [ ] 已证明没有两个 PostgreSQL 实例同时挂载同一 PGDATA；任何不一致均停止发布
|
||||
- [ ] 已记录活动 server/db 容器、不可变镜像 ID（及可用 repo digest）、Caddy 目标、挂载源、环境摘要，不记录 env 正文
|
||||
- [ ] 已冻结成功 migration 数量（当前预期 58）和关键业务表逐表计数基线
|
||||
- [ ] 已现场只读冻结成功 migration 数量和关键业务表逐表计数基线；当前生产证据为 61 项、本地为 62 项，本次待上线只应新增 1 项，现场结果不符立即停止
|
||||
- [ ] 确认当前版本已打 tag：`git tag -a trialrun-YYYYMMDD-NN -m "trialrun release"`
|
||||
- [ ] 本地门禁：`./scripts/check-minimal.sh`
|
||||
- [ ] 部署契约校验：`node deploy/trialrun/validate-env-contract.js`
|
||||
@@ -48,12 +48,14 @@
|
||||
- [ ] 完整批次已复制到获批异机/异盘，并记录保留周期
|
||||
- [ ] 同批次已在隔离影子目标恢复：migration 数、关键业务表逐表计数、附件普通文件数均与冻结基线一致，兼容镜像 `/ready` 通过
|
||||
- [ ] 自动异机备份（未完成远端门禁前保持未勾选）：
|
||||
- [ ] **EXTERNAL_GATED**：ARM 已获批安装 rsync、密码锁定专用账号、forced-command key 和独立 inbox/archive/weekly；普通 shell、rsync sender/read/delete 均被拒绝
|
||||
- [ ] **EXTERNAL_GATED**：生产专用私钥为 `0600`，known-hosts 指纹已人工核对，安装器显式绑定活动 project、PGDATA、附件、完整/短构建 SHA、备份/状态目录和远端身份
|
||||
- [ ] **EXTERNAL_GATED**：用户已在独立企微运维群创建机器人，并分别把 Webhook 写入生产、ARM 的 root-only `0600` 文件；未向 Codex、仓库、普通 `.env`、日志或聊天提供正文
|
||||
- [ ] **EXTERNAL_GATED**：ARM 已获批安装 rsync、密码锁定专用账号、forced-command key、独立 inbox/archive/weekly 和 26 小时 dead-man；普通 shell、rsync sender/read/delete 均被拒绝
|
||||
- [ ] **EXTERNAL_GATED**：生产专用私钥为 `0600`，known-hosts 指纹已人工核对，安装器显式绑定活动 project、PGDATA、附件、完整/短构建 SHA、备份/状态目录、远端身份和 Webhook 文件路径
|
||||
- [ ] `install-automatic-backup-timer.sh install` 只安装 disabled units；首批真实批次及 ARM manifest/SHA-256 核对通过后才显式执行 `enable`
|
||||
- [ ] daily/watchdog/drill-reminder 三组 timer enabled/active，daily service 当前 inactive；未通过手工启动真实告警来伪造验证
|
||||
- [ ] 生产 daily/watchdog/drill-reminder 与 ARM dead-man 四组 timer enabled/active，各 oneshot service 当前 inactive；未通过只调用机器人发送入口来伪造验证
|
||||
- [ ] 最近一次异机成功不超过 24 小时；生产自动日批次为 7 个，ARM 自动日批次为 7 个、周检查点为 4 个，历史手工批次未被轮转
|
||||
- [ ] **EXTERNAL_GATED**：可控传输失败仅产生一条脱敏企微告警，恢复链路后新批次成功；同一失败重复运行已降噪
|
||||
- [ ] **EXTERNAL_GATED**：已真正停止一次源端 timer 或故意破坏一次传输，ARM dead-man 仅产生一条脱敏企微告警；恢复 finalize 后收到一次恢复通知，同一失败重复运行已降噪
|
||||
- [ ] 若调整过 `BACKUP_RUN_AT`，已人工确认调整后的次日 UTC 分桶和 26 小时窗口，未把一次预期空窗误判为长期故障
|
||||
- [ ] 本季度已由第二名内部人员按 runbook 首部恢复卡在 ARM 隔离目标完成影子恢复走查；timer 未自动恢复生产
|
||||
- [ ] 停机复制方案已冻结：旧 PostgreSQL 完全停止后复制到空新目录并保留所有权/权限/链接/扩展属性；旧目录原位保留且观察期内不再写入
|
||||
|
||||
|
||||
@@ -14,7 +14,8 @@ AUTOMATIC_BACKUP_STATE_DIR="${AUTOMATIC_BACKUP_STATE_DIR:-}"
|
||||
OFFHOST_TARGET="${OFFHOST_TARGET:-}"
|
||||
OFFHOST_SSH_KEY="${OFFHOST_SSH_KEY:-}"
|
||||
OFFHOST_KNOWN_HOSTS="${OFFHOST_KNOWN_HOSTS:-}"
|
||||
ALERT_RECIPIENT_EMPLOYEE_NO="${ALERT_RECIPIENT_EMPLOYEE_NO:-HYD000006}"
|
||||
ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
|
||||
ALERT_HELPER="${ALERT_HELPER:-${REMOTE_DIR}/scripts/send-wecom-group-alert.py}"
|
||||
BACKUP_RUN_AT="${BACKUP_RUN_AT:-02:30}"
|
||||
WATCHDOG_RUN_AT="${WATCHDOG_RUN_AT:-07:10}"
|
||||
SERVICE_PREFIX="${SERVICE_PREFIX:-hyb-oa-automatic-backup}"
|
||||
@@ -66,7 +67,8 @@ paths_overlap() {
|
||||
validate_inputs() {
|
||||
local name value
|
||||
for name in REMOTE_DIR REMOTE_ENV_FILE POSTGRES_DATA_DIR STORAGE_HOST_DIR \
|
||||
AUTOMATIC_BACKUP_DIR AUTOMATIC_BACKUP_STATE_DIR OFFHOST_SSH_KEY OFFHOST_KNOWN_HOSTS; do
|
||||
AUTOMATIC_BACKUP_DIR AUTOMATIC_BACKUP_STATE_DIR OFFHOST_SSH_KEY OFFHOST_KNOWN_HOSTS \
|
||||
ALERT_WEBHOOK_FILE ALERT_HELPER; do
|
||||
value="${!name:-}"
|
||||
validate_absolute_path "$name" "$value"
|
||||
done
|
||||
@@ -91,10 +93,14 @@ validate_inputs() {
|
||||
&& "$(stat -c %a "$OFFHOST_SSH_KEY")" == "600" ]] \
|
||||
|| fail "off-host private key must exist with mode 0600"
|
||||
[[ -s "$OFFHOST_KNOWN_HOSTS" ]] || fail "pinned known-hosts file is missing"
|
||||
[[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
|
||||
&& "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" \
|
||||
&& "$(stat -c %u "$ALERT_WEBHOOK_FILE")" == "0" ]] \
|
||||
|| fail "alert webhook file must be root-owned with mode 0600"
|
||||
[[ -f "$ALERT_HELPER" && ! -L "$ALERT_HELPER" ]] || fail "alert helper is missing"
|
||||
[[ "$BACKUP_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "backup time is invalid"
|
||||
[[ "$WATCHDOG_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "watchdog time is invalid"
|
||||
[[ "$SERVICE_PREFIX" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "service prefix is invalid"
|
||||
[[ "$ALERT_RECIPIENT_EMPLOYEE_NO" =~ ^[A-Za-z0-9_-]{1,80}$ ]] || fail "alert recipient id is invalid"
|
||||
local -a isolated_paths=(
|
||||
"$POSTGRES_DATA_DIR" "$STORAGE_HOST_DIR" "$AUTOMATIC_BACKUP_DIR" "$AUTOMATIC_BACKUP_STATE_DIR"
|
||||
)
|
||||
@@ -123,7 +129,8 @@ Environment=AUTOMATIC_BACKUP_STATE_DIR=${AUTOMATIC_BACKUP_STATE_DIR}
|
||||
Environment=OFFHOST_TARGET=${OFFHOST_TARGET}
|
||||
Environment=OFFHOST_SSH_KEY=${OFFHOST_SSH_KEY}
|
||||
Environment=OFFHOST_KNOWN_HOSTS=${OFFHOST_KNOWN_HOSTS}
|
||||
Environment=ALERT_RECIPIENT_EMPLOYEE_NO=${ALERT_RECIPIENT_EMPLOYEE_NO}
|
||||
Environment=ALERT_WEBHOOK_FILE=${ALERT_WEBHOOK_FILE}
|
||||
Environment=ALERT_HELPER=${ALERT_HELPER}
|
||||
ENVIRONMENT
|
||||
}
|
||||
|
||||
|
||||
@@ -7,6 +7,15 @@ PUBLIC_KEY_FILE="${PUBLIC_KEY_FILE:-}"
|
||||
RECEIVER_SOURCE="${RECEIVER_SOURCE:-}"
|
||||
RECEIVER_TARGET="${RECEIVER_TARGET:-/usr/local/sbin/hyb-oa-offhost-receiver}"
|
||||
CONFIG_FILE="${CONFIG_FILE:-/etc/hyb-oa-offhost-backup.conf}"
|
||||
ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
|
||||
ALERT_HELPER_SOURCE="${ALERT_HELPER_SOURCE:-}"
|
||||
ALERT_HELPER_TARGET="${ALERT_HELPER_TARGET:-/usr/local/sbin/hyb-oa-wecom-group-alert}"
|
||||
DEADMAN_SOURCE="${DEADMAN_SOURCE:-}"
|
||||
DEADMAN_TARGET="${DEADMAN_TARGET:-/usr/local/sbin/hyb-oa-offhost-deadman}"
|
||||
DEADMAN_MAX_AGE_SECONDS="${DEADMAN_MAX_AGE_SECONDS:-93600}"
|
||||
DEADMAN_RUN_AT="${DEADMAN_RUN_AT:-06:30}"
|
||||
DEADMAN_SERVICE="${DEADMAN_SERVICE:-hyb-oa-offhost-deadman}"
|
||||
SYSTEMD_DIR="${SYSTEMD_DIR:-/etc/systemd/system}"
|
||||
DAILY_KEEP="${DAILY_KEEP:-7}"
|
||||
WEEKLY_KEEP="${WEEKLY_KEEP:-4}"
|
||||
MODE="${1:-install}"
|
||||
@@ -45,7 +54,7 @@ validate_backup_root() {
|
||||
if [[ -d "$BACKUP_ROOT" ]]; then
|
||||
local unmanaged_entry
|
||||
unmanaged_entry="$(find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 \
|
||||
! -name inbox ! -name archive ! -name weekly -print -quit)"
|
||||
! -name inbox ! -name archive ! -name weekly ! -name state -print -quit)"
|
||||
[[ -z "$unmanaged_entry" ]] || fail "backup root contains an unmanaged top-level entry"
|
||||
fi
|
||||
}
|
||||
@@ -57,8 +66,22 @@ validate_inputs() {
|
||||
validate_absolute_path RECEIVER_SOURCE "$RECEIVER_SOURCE"
|
||||
validate_absolute_path RECEIVER_TARGET "$RECEIVER_TARGET"
|
||||
validate_absolute_path CONFIG_FILE "$CONFIG_FILE"
|
||||
validate_absolute_path ALERT_WEBHOOK_FILE "$ALERT_WEBHOOK_FILE"
|
||||
validate_absolute_path ALERT_HELPER_SOURCE "$ALERT_HELPER_SOURCE"
|
||||
validate_absolute_path ALERT_HELPER_TARGET "$ALERT_HELPER_TARGET"
|
||||
validate_absolute_path DEADMAN_SOURCE "$DEADMAN_SOURCE"
|
||||
validate_absolute_path DEADMAN_TARGET "$DEADMAN_TARGET"
|
||||
[[ -f "$PUBLIC_KEY_FILE" && ! -L "$PUBLIC_KEY_FILE" ]] || fail "public key file is missing"
|
||||
[[ -f "$RECEIVER_SOURCE" && ! -L "$RECEIVER_SOURCE" ]] || fail "receiver source is missing"
|
||||
[[ -f "$ALERT_HELPER_SOURCE" && ! -L "$ALERT_HELPER_SOURCE" ]] || fail "alert helper source is missing"
|
||||
[[ -f "$DEADMAN_SOURCE" && ! -L "$DEADMAN_SOURCE" ]] || fail "dead-man source is missing"
|
||||
[[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
|
||||
&& "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" \
|
||||
&& "$(stat -c %u "$ALERT_WEBHOOK_FILE")" == "0" ]] \
|
||||
|| fail "alert webhook file must be root-owned with mode 0600"
|
||||
[[ "$DEADMAN_MAX_AGE_SECONDS" =~ ^[1-9][0-9]*$ ]] || fail "dead-man threshold must be positive"
|
||||
[[ "$DEADMAN_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "dead-man time is invalid"
|
||||
[[ "$DEADMAN_SERVICE" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "dead-man service name is invalid"
|
||||
[[ "$DAILY_KEEP" =~ ^[1-9][0-9]*$ && "$WEEKLY_KEEP" =~ ^[1-9][0-9]*$ ]] \
|
||||
|| fail "retention values must be positive"
|
||||
command -v rsync >/dev/null 2>&1 || fail "rsync must be installed before receiver setup"
|
||||
@@ -86,22 +109,28 @@ install_receiver() {
|
||||
fi
|
||||
passwd -l "$BACKUP_USER" >/dev/null 2>&1 || true
|
||||
|
||||
local home_dir inbox_dir archive_dir weekly_dir ssh_dir auth_file public_key rrsync_bin
|
||||
local home_dir inbox_dir archive_dir weekly_dir state_dir ssh_dir auth_file public_key rrsync_bin
|
||||
home_dir="$(getent passwd "$BACKUP_USER" | awk -F: '{print $6}')"
|
||||
validate_absolute_path BACKUP_USER_HOME "$home_dir"
|
||||
inbox_dir="$BACKUP_ROOT/inbox"
|
||||
archive_dir="$BACKUP_ROOT/archive"
|
||||
weekly_dir="$BACKUP_ROOT/weekly"
|
||||
mkdir -p "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh"
|
||||
state_dir="$BACKUP_ROOT/state"
|
||||
mkdir -p "$inbox_dir" "$archive_dir" "$weekly_dir" "$state_dir" "$home_dir/.ssh"
|
||||
[[ -d "$BACKUP_ROOT" && ! -L "$BACKUP_ROOT" \
|
||||
&& -d "$inbox_dir" && ! -L "$inbox_dir" \
|
||||
&& -d "$archive_dir" && ! -L "$archive_dir" \
|
||||
&& -d "$weekly_dir" && ! -L "$weekly_dir" ]] \
|
||||
|| fail "backup receiver directories must not be symbolic links"
|
||||
chown -R "$BACKUP_USER:$BACKUP_USER" "$BACKUP_ROOT" "$home_dir/.ssh"
|
||||
chown "$BACKUP_USER:$BACKUP_USER" "$BACKUP_ROOT"
|
||||
chown -R "$BACKUP_USER:$BACKUP_USER" "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh"
|
||||
chmod 0700 "$BACKUP_ROOT" "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh"
|
||||
chown -R root:root "$state_dir"
|
||||
chmod 0700 "$state_dir"
|
||||
|
||||
install -o root -g root -m 0755 "$RECEIVER_SOURCE" "$RECEIVER_TARGET"
|
||||
install -o root -g root -m 0755 "$ALERT_HELPER_SOURCE" "$ALERT_HELPER_TARGET"
|
||||
install -o root -g root -m 0755 "$DEADMAN_SOURCE" "$DEADMAN_TARGET"
|
||||
rrsync_bin="$(command -v rrsync)"
|
||||
cat >"$CONFIG_FILE" <<CONFIG
|
||||
BACKUP_INBOX_DIR=${inbox_dir}
|
||||
@@ -130,7 +159,37 @@ AUTH
|
||||
|
||||
runuser -u "$BACKUP_USER" -- env RECEIVER_CONFIG_FILE="$CONFIG_FILE" \
|
||||
"$RECEIVER_TARGET" validate-config >/dev/null
|
||||
echo "[offhost-receiver-installer][PASS] restricted receiver installed"
|
||||
cat >"$SYSTEMD_DIR/${DEADMAN_SERVICE}.service" <<SERVICE
|
||||
[Unit]
|
||||
Description=hyb-oa independent off-host backup dead-man
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
UMask=0077
|
||||
ExecStart=/usr/bin/python3 ${DEADMAN_TARGET} --archive-root ${archive_dir} --state-dir ${state_dir} --webhook-file ${ALERT_WEBHOOK_FILE} --alert-helper ${ALERT_HELPER_TARGET} --max-age-seconds ${DEADMAN_MAX_AGE_SECONDS}
|
||||
SERVICE
|
||||
cat >"$SYSTEMD_DIR/${DEADMAN_SERVICE}.timer" <<TIMER
|
||||
[Unit]
|
||||
Description=Check hyb-oa off-host backup freshness independently
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* ${DEADMAN_RUN_AT}:00
|
||||
Persistent=true
|
||||
Unit=${DEADMAN_SERVICE}.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
TIMER
|
||||
systemctl daemon-reload
|
||||
echo "[offhost-receiver-installer][PASS] restricted receiver and disabled dead-man installed"
|
||||
}
|
||||
|
||||
enable_receiver() {
|
||||
install_receiver
|
||||
systemctl enable --now "${DEADMAN_SERVICE}.timer"
|
||||
echo "[offhost-receiver-installer][PASS] dead-man timer enabled"
|
||||
}
|
||||
|
||||
uninstall_receiver() {
|
||||
@@ -138,6 +197,12 @@ uninstall_receiver() {
|
||||
[[ "$BACKUP_USER" =~ ^[a-z_][a-z0-9_-]{0,30}$ ]] || fail "backup user is invalid"
|
||||
validate_absolute_path RECEIVER_TARGET "$RECEIVER_TARGET"
|
||||
validate_absolute_path CONFIG_FILE "$CONFIG_FILE"
|
||||
validate_absolute_path ALERT_HELPER_TARGET "$ALERT_HELPER_TARGET"
|
||||
validate_absolute_path DEADMAN_TARGET "$DEADMAN_TARGET"
|
||||
[[ "$DEADMAN_SERVICE" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "dead-man service name is invalid"
|
||||
systemctl disable --now "${DEADMAN_SERVICE}.timer" >/dev/null 2>&1 || true
|
||||
systemctl stop "${DEADMAN_SERVICE}.service" >/dev/null 2>&1 || true
|
||||
rm -f -- "$SYSTEMD_DIR/${DEADMAN_SERVICE}.timer" "$SYSTEMD_DIR/${DEADMAN_SERVICE}.service"
|
||||
if id "$BACKUP_USER" >/dev/null 2>&1; then
|
||||
local home_dir auth_file
|
||||
home_dir="$(getent passwd "$BACKUP_USER" | awk -F: '{print $6}')"
|
||||
@@ -146,19 +211,23 @@ uninstall_receiver() {
|
||||
rm -f -- "$auth_file"
|
||||
fi
|
||||
fi
|
||||
rm -f -- "$RECEIVER_TARGET" "$CONFIG_FILE"
|
||||
echo "[offhost-receiver-installer][PASS] receiver authorization removed; archives retained"
|
||||
rm -f -- "$RECEIVER_TARGET" "$CONFIG_FILE" "$ALERT_HELPER_TARGET" "$DEADMAN_TARGET"
|
||||
systemctl daemon-reload
|
||||
echo "[offhost-receiver-installer][PASS] receiver authorization and dead-man removed; archives retained"
|
||||
}
|
||||
|
||||
case "$MODE" in
|
||||
install)
|
||||
install_receiver
|
||||
;;
|
||||
enable)
|
||||
enable_receiver
|
||||
;;
|
||||
uninstall|--uninstall)
|
||||
uninstall_receiver
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: BACKUP_ROOT=<absolute> PUBLIC_KEY_FILE=<absolute> RECEIVER_SOURCE=<absolute> bash install-offhost-backup-receiver.sh [install|uninstall]"
|
||||
echo "Usage: <explicit backup/key/helper/dead-man/webhook paths> bash install-offhost-backup-receiver.sh [install|enable|uninstall]"
|
||||
;;
|
||||
*)
|
||||
fail "unsupported mode"
|
||||
|
||||
+23
-6
@@ -226,29 +226,46 @@ manifest 必须至少记录一个真实来源标识：活动 `/ready` 有构建
|
||||
|
||||
整批复制到获批的异机/异盘位置并记录保留周期；不能只复制 DB 或附件中的一份。
|
||||
|
||||
#### 每日自动异机备份（当前仅本地实现，远端启用为 EXTERNAL_GATED）
|
||||
#### 每日自动异机备份（可靠启用闭环仍为 EXTERNAL_GATED）
|
||||
|
||||
- 每日 02:30 由生产主机解析唯一活动 Compose project 和真实 DB/附件挂载，生成同批三文件，ARM 校验并原子归档成功后才写成功状态和执行源端轮转。
|
||||
- 生产保留 7 个自动日批次；ARM 保留 7 个日批次和 4 个周检查点。07:10 watchdog 检查最近 24 小时异机成功记录；季度 timer 只发送人工影子恢复提醒。
|
||||
- 生产保留 7 个自动日批次；ARM 保留 7 个日批次和 4 个周检查点。07:10 源端 watchdog 检查最近 24 小时异机成功记录；ARM 独立 dead-man 检查最近 26 小时归档，生产整机或 OA server 停止时仍能告警；季度 timer 只发送人工影子恢复提醒。
|
||||
- ARM 使用专用密码锁定账号和 forced-command key：rsync 只能写 inbox，不能读取/删除 archive，也没有普通 shell。生产固定专用 key 和人工核对的 known-hosts，禁止 `accept-new`。
|
||||
- 失败告警只包含阶段、状态、finding 数和类别建议。同一失败降噪；首次失败、状态恶化、新 finding 和每季度首次恢复提醒才发送。
|
||||
- 同一 UTC 日重试使用新 run id 时，ARM 必须先完整校验候选，再原子取代当日旧批次并重建该周硬链接检查点；坏候选或发布失败不得触碰原有效批次。
|
||||
- 生产与 ARM 都直接使用独立企微运维群机器人，不依赖 OA server、数据库或个人工号。Webhook 正文只由用户手工写入两台主机的 root-only `0600` 文件；仓库、普通 `.env`、systemd unit、日志和聊天只能出现受控文件路径，不能出现 URL 或 key。
|
||||
- 失败告警只包含阶段、状态、finding 数和类别建议；原始命令诊断只保留在源端状态目录的 `0600` 最近失败日志。同一失败降噪；首次失败、状态恶化、新 finding、恢复通知和每季度首次恢复提醒才发送。新部署构建与最近恢复点 build SHA 不同只记 finding，缺失、损坏、超龄、未来时间或 Compose project 不同仍硬失败。
|
||||
|
||||
创建机器人后，由用户分别登录两台主机，在 root shell 中用静默输入把完整 Webhook 写入受控路径；不要把值作为命令参数、环境文件内容或聊天消息交给 Codex：
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
read -rsp 'WeCom operations robot Webhook: ' HYB_OA_ALERT_WEBHOOK
|
||||
printf '\n'
|
||||
install -o root -g root -m 0600 /dev/null <controlled-webhook-file>
|
||||
printf '%s\n' "$HYB_OA_ALERT_WEBHOOK" > <controlled-webhook-file>
|
||||
unset HYB_OA_ALERT_WEBHOOK
|
||||
```
|
||||
|
||||
安装顺序必须是：先获批 ARM 端安装 rsync、受限身份和 receiver，用合成批次验证；再获批生产端安装 units，手工完成首个真实批次；最后显式执行生产安装器的 `enable`。两个安装器默认 `install` 均不代表已启用，不得把本地合同通过写成远端完成。
|
||||
|
||||
```bash
|
||||
# ARM：参数从受控配置取得；install 会写账号/authorized_keys/receiver，需单独批准。
|
||||
BACKUP_ROOT=<absolute-backup-root> PUBLIC_KEY_FILE=<dedicated-public-key> \
|
||||
RECEIVER_SOURCE=<repo-receiver-script> \
|
||||
RECEIVER_SOURCE=<repo-receiver-script> ALERT_WEBHOOK_FILE=<controlled-webhook-file> \
|
||||
ALERT_HELPER_SOURCE=<repo-group-alert-helper> DEADMAN_SOURCE=<repo-deadman-script> \
|
||||
bash deploy/trialrun/install-offhost-backup-receiver.sh install
|
||||
|
||||
# 生产：先只安装 disabled units；路径、项目、构建和 SSH 身份必须绑定活动实例。
|
||||
<explicit-project-data-build-and-ssh-env> \
|
||||
<explicit-project-data-build-ssh-and-webhook-file-env> \
|
||||
bash deploy/trialrun/install-automatic-backup-timer.sh install
|
||||
|
||||
# 首批真实备份、ARM 哈希和脱敏告警均经单独确认后，才允许启用三组 timer。
|
||||
<same-arm-explicit-env> bash deploy/trialrun/install-offhost-backup-receiver.sh enable
|
||||
<same-explicit-env> bash deploy/trialrun/install-automatic-backup-timer.sh enable
|
||||
```
|
||||
|
||||
`BACKUP_RUN_AT` 使用生产主机本地时区，ARM 归档按 UTC 日分桶。调整执行时刻的当天可能出现预期空窗；改动后必须人工确认次日 UTC 分桶、下一次 daily 运行和 26 小时 dead-man 窗口。真实告警验收必须真正停止一次源端 timer 或故意让一次传输失败，再确认 ARM 首次告警、重复降噪和恢复通知；只测试机器人能发消息不算通过。
|
||||
|
||||
卸载只撤销 timer/receiver 授权，保留源端批次、状态、ARM archive 和凭证文件；专用账号/密钥的最终销毁另走获批清理。
|
||||
|
||||
### 1.5.2 本地隔离恢复演练
|
||||
@@ -284,7 +301,7 @@ RESTORE_SCOPE=shadow \
|
||||
|
||||
`restore-db.sh` 和 `restore-storage.sh` 默认只接受 `shadow`。影子库名必须包含独立的 `shadow`、`drill` 或 `restore_test` 标记；数据库确认不匹配、库名不安全、附件目录非空、目标为默认试运行附件目录或归档路径危险时均在写入前退出。
|
||||
|
||||
真实格式影子恢复的验收不仅是 `/ready`：恢复后的 `_prisma_migrations` 成功项必须等于冻结基线（当前预期 58），预先列出的关键业务表计数必须逐表一致，附件解包后的普通文件数必须与活动源目录一致。任一项不一致即判定批次不可作为切换回滚点。
|
||||
真实格式影子恢复的验收不仅是 `/ready`：恢复后的 `_prisma_migrations` 成功项必须等于执行前现场冻结基线（当前生产证据为 61 项；本地 62 项仅表示下一次部署预计新增 1 项），预先列出的关键业务表计数必须逐表一致，附件解包后的普通文件数必须与活动源目录一致。任一项不一致即判定批次不可作为切换回滚点。
|
||||
|
||||
### 1.5.3 生产切换前冻结记录
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#!/usr/bin/env node
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const childProcess = require('node:child_process');
|
||||
|
||||
const rootDir = path.resolve(__dirname, '..', '..');
|
||||
const configPath = path.join(rootDir, 'server/src/config.js');
|
||||
@@ -29,6 +30,8 @@ const offhostReceiverInstallerPath = path.join(
|
||||
);
|
||||
const automaticBackupRunnerPath = path.join(rootDir, 'scripts/run-automatic-backup.sh');
|
||||
const offhostReceiverPath = path.join(rootDir, 'scripts/receive-offhost-backup.sh');
|
||||
const groupAlertHelperPath = path.join(rootDir, 'scripts/send-wecom-group-alert.py');
|
||||
const offhostDeadmanPath = path.join(rootDir, 'scripts/check-offhost-backup-deadman.py');
|
||||
const nginxPaths = [
|
||||
path.join(rootDir, 'deploy/trialrun/nginx-hyb-oa.conf'),
|
||||
path.join(rootDir, 'deploy/trialrun/nginx.conf.example'),
|
||||
@@ -157,6 +160,8 @@ const automaticBackupTimerScriptContent = readFile(automaticBackupTimerScriptPat
|
||||
const offhostReceiverInstallerContent = readFile(offhostReceiverInstallerPath);
|
||||
const automaticBackupRunnerContent = readFile(automaticBackupRunnerPath);
|
||||
const offhostReceiverContent = readFile(offhostReceiverPath);
|
||||
const groupAlertHelperContent = readFile(groupAlertHelperPath);
|
||||
const offhostDeadmanContent = readFile(offhostDeadmanPath);
|
||||
const nginxContents = nginxPaths.map((filePath) => ({
|
||||
filePath,
|
||||
content: readFile(filePath),
|
||||
@@ -185,6 +190,25 @@ for (const key of ['APP_HOST_PORT', 'POSTGRES_DATA_DIR', 'STORAGE_HOST_DIR']) {
|
||||
if (envExampleKeys.has('COMPOSE_PROJECT_NAME')) {
|
||||
errors.push('.env.example 不应持久化 COMPOSE_PROJECT_NAME，调用方必须通过 -p 显式选择项目');
|
||||
}
|
||||
for (const key of ['ALERT_WEBHOOK_FILE', 'WECOM_WEBHOOK_URL', 'WECOM_WEBHOOK_KEY']) {
|
||||
if (envExampleKeys.has(key)) {
|
||||
errors.push(`.env.example 不应持久化企微群机器人配置 ${key}`);
|
||||
}
|
||||
}
|
||||
const trackedFiles = childProcess.execFileSync('git', ['ls-files', '--cached', '--others', '--exclude-standard', '-z'], {
|
||||
cwd: rootDir,
|
||||
encoding: 'utf8',
|
||||
}).split('\0').filter(Boolean);
|
||||
const literalWebhookPattern = /https:\/\/qyapi\.weixin\.qq\.com\/cgi-bin\/webhook\/send\?key=/;
|
||||
for (const relativePath of trackedFiles) {
|
||||
const filePath = path.join(rootDir, relativePath);
|
||||
if (!fs.statSync(filePath).isFile()) continue;
|
||||
const content = fs.readFileSync(filePath);
|
||||
if (content.includes(0)) continue;
|
||||
if (literalWebhookPattern.test(content.toString('utf8'))) {
|
||||
errors.push(`${relativePath} 不得持久化完整企微群机器人 Webhook URL`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const serviceName of requiredServices) {
|
||||
const block = extractServiceBlock(composeContent, serviceName);
|
||||
@@ -306,6 +330,8 @@ for (const requiredFragment of [
|
||||
'Environment=OFFHOST_TARGET=${OFFHOST_TARGET}',
|
||||
'Environment=OFFHOST_SSH_KEY=${OFFHOST_SSH_KEY}',
|
||||
'Environment=OFFHOST_KNOWN_HOSTS=${OFFHOST_KNOWN_HOSTS}',
|
||||
'Environment=ALERT_WEBHOOK_FILE=${ALERT_WEBHOOK_FILE}',
|
||||
'Environment=ALERT_HELPER=${ALERT_HELPER}',
|
||||
'install)\n install_units',
|
||||
'enable)\n enable_units',
|
||||
]) {
|
||||
@@ -325,7 +351,10 @@ for (const requiredFragment of [
|
||||
'hyb-backup-finalize $run_id $manifest_sha',
|
||||
'write_success_state "$run_id" "$manifest_sha"',
|
||||
'prune_local_backups "$batch_name"',
|
||||
'--summary-file=-',
|
||||
'python3 "$ALERT_HELPER"',
|
||||
'--webhook-file "$ALERT_WEBHOOK_FILE"',
|
||||
'capture_command offhost_transfer rsync',
|
||||
'last-error.log',
|
||||
"docker inspect -f '{{json .Mounts}}'",
|
||||
'if len(matches) != 1 or not matches[0]:',
|
||||
]) {
|
||||
@@ -345,6 +374,8 @@ for (const requiredFragment of [
|
||||
'batch must contain exactly three regular files',
|
||||
'manifest artifact checksum mismatch',
|
||||
'mv -- "$partial_dir" "$archive_dir"',
|
||||
'replacement_rollback=',
|
||||
'publish_weekly_checkpoint',
|
||||
'prune_named_directories "$BACKUP_ARCHIVE_DIR"',
|
||||
'prune_named_directories "$BACKUP_WEEKLY_DIR"',
|
||||
]) {
|
||||
@@ -360,6 +391,9 @@ for (const requiredFragment of [
|
||||
'BACKUP_INBOX_DIR=${inbox_dir}',
|
||||
'BACKUP_ARCHIVE_DIR=${archive_dir}',
|
||||
'BACKUP_WEEKLY_DIR=${weekly_dir}',
|
||||
'DEADMAN_MAX_AGE_SECONDS="${DEADMAN_MAX_AGE_SECONDS:-93600}"',
|
||||
'ExecStart=/usr/bin/python3 ${DEADMAN_TARGET}',
|
||||
'systemctl enable --now "${DEADMAN_SERVICE}.timer"',
|
||||
'archives retained',
|
||||
'backup root contains an unmanaged top-level entry',
|
||||
'backup root must not be inside a system-managed directory',
|
||||
@@ -369,6 +403,30 @@ for (const requiredFragment of [
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'stat.S_IMODE(info.st_mode) != 0o600',
|
||||
'parsed.hostname != "qyapi.weixin.qq.com"',
|
||||
'HYB_WECOM_ALERT_TEST_MODE',
|
||||
'should_send(previous, status, category)',
|
||||
'不要在群内粘贴原始日志',
|
||||
]) {
|
||||
if (!groupAlertHelperContent.includes(requiredFragment)) {
|
||||
errors.push(`send-wecom-group-alert.py 缺少独立脱敏告警合同: ${requiredFragment}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'default=93600',
|
||||
'archive_manifest_invalid',
|
||||
'archive_time_in_future',
|
||||
'FAIL_OFFHOST_DEADMAN',
|
||||
'deadman-alert-state.json',
|
||||
]) {
|
||||
if (!offhostDeadmanContent.includes(requiredFragment)) {
|
||||
errors.push(`check-offhost-backup-deadman.py 缺少 ARM 独立看门狗合同: ${requiredFragment}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'validate_data_path POSTGRES_DATA_DIR "$POSTGRES_DATA_DIR"',
|
||||
'validate_data_path STORAGE_HOST_DIR "$STORAGE_HOST_DIR"',
|
||||
|
||||
+4
-3
@@ -359,12 +359,13 @@ WAPC2 请假/补卡只读 POC 操作口径：
|
||||
自动异机备份运维口径：
|
||||
|
||||
1. 自动备份以 OA 仓库 runbook 首部“紧急恢复卡”为唯一短入口；公司知识库保存该卡副本，并指定系统管理员和一名替补操作人。
|
||||
2. 每日 02:30 生成数据库、附件、manifest 同批恢复点并推送 ARM；07:10 检查最近 24 小时异机成功状态。生产保留 7 个自动日批次，ARM 保留 7 个日批次和 4 个周检查点。
|
||||
2. 每日 02:30 生成数据库、附件、manifest 同批恢复点并推送 ARM；07:10 检查最近 24 小时异机成功状态，ARM 另以 26 小时 dead-man 独立检查归档。生产保留 7 个自动日批次，ARM 保留 7 个日批次和 4 个周检查点。
|
||||
3. ARM 专用身份只能写 inbox 和请求 finalize，不能登录 shell、读取或删除 archive。任何人不得为了省事改用普通 ARM 登录账号或关闭 known-hosts 固定校验。
|
||||
4. 企微只收脱敏异常：首次失败、恶化或新 finding 发送；同一失败降噪。季度提醒只要求人工在 ARM 隔离目标恢复，不会自动恢复生产。
|
||||
5. 当前仓库实现和本地合成合同不等于服务器已安装。ARM receiver、生产 units、首次真实批次、timer enable 和真实失败告警均为 **EXTERNAL_GATED**，必须逐阶段批准并回填证据。
|
||||
4. 自动备份使用独立企微运维群机器人，不依赖 OA server 或个人员工号。Webhook 只由用户写入两台主机的 root-only 文件，不进仓库、普通 `.env`、日志或聊天。企微只收脱敏异常、finding 和恢复通知；同一失败降噪。季度提醒只要求人工在 ARM 隔离目标恢复，不会自动恢复生产。
|
||||
5. 当前 receiver 和生产 units 的既有安装不等于可靠启用闭环完成。群机器人、ARM dead-man、更新后的两端脚本、首次有效真实批次、全部 timer enable 和真实断链告警均为 **EXTERNAL_GATED**，必须逐阶段批准并回填证据；真实验收必须停止一次源端 timer 或故意让一次传输失败，不能只测试机器人发送。
|
||||
6. 每季度由替补操作人按恢复卡走查，至少能指出归档类别、三条恢复入口、`/ready` 和关键计数验收项。不能独立说明时，本季度恢复门禁不通过。
|
||||
7. 卸载 timer 或 receiver 时保留历史 archive 和源端完整批次；账号、密钥、备份数据的删除必须另行批准。
|
||||
8. 改动每日备份时刻后，人工确认次日 UTC 分桶和 26 小时窗口，避免把调整日的预期空窗误当成长期故障。
|
||||
|
||||
WROAD2/WARCH3 企微受控上线每日巡检：
|
||||
|
||||
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate the newest ARM archive and drive an independent dead-man alert."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
BATCH_RE = re.compile(r"^hyb-oa-backup-(auto-(\d{8}T\d{6}Z))$")
|
||||
|
||||
|
||||
def fail(message: str) -> "NoReturn":
|
||||
print(f"[offhost-deadman][ERROR] {message}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def validate_latest(archive_root: Path, max_age: int) -> tuple[str, str]:
|
||||
if not archive_root.is_dir() or archive_root.is_symlink():
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_root_invalid"
|
||||
candidates = sorted(
|
||||
(entry for entry in archive_root.iterdir() if entry.is_dir() and not entry.is_symlink() and BATCH_RE.fullmatch(entry.name)),
|
||||
key=lambda entry: entry.name,
|
||||
)
|
||||
if not candidates:
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_missing"
|
||||
batch = candidates[-1]
|
||||
match = BATCH_RE.fullmatch(batch.name)
|
||||
assert match is not None
|
||||
run_id, stamp = match.groups()
|
||||
expected_names = {"database.dump", "storage.tar.gz", "manifest.json"}
|
||||
entries = list(batch.iterdir())
|
||||
if {entry.name for entry in entries} != expected_names or any(
|
||||
entry.is_symlink() or not stat.S_ISREG(entry.stat().st_mode) for entry in entries
|
||||
):
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_files_invalid"
|
||||
try:
|
||||
manifest = json.loads((batch / "manifest.json").read_text(encoding="utf-8"))
|
||||
if manifest.get("schemaVersion") != 1 or manifest.get("status") != "complete" or manifest.get("runId") != run_id:
|
||||
raise ValueError
|
||||
artifacts = manifest.get("artifacts") or {}
|
||||
for key, name in (("database", "database.dump"), ("storage", "storage.tar.gz")):
|
||||
item = artifacts.get(key) or {}
|
||||
target = batch / name
|
||||
if item.get("file") != name or item.get("bytes") != target.stat().st_size or item.get("sha256") != sha256(target):
|
||||
raise ValueError
|
||||
completed = datetime.strptime(stamp, "%Y%m%dT%H%M%SZ").replace(tzinfo=timezone.utc)
|
||||
except (OSError, ValueError, TypeError, json.JSONDecodeError):
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_manifest_invalid"
|
||||
age = (datetime.now(timezone.utc) - completed).total_seconds()
|
||||
if age < 0:
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_time_in_future"
|
||||
if age > max_age:
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_stale"
|
||||
return "PASS_OFFHOST_DEADMAN_FRESH", "fresh_offhost_archive"
|
||||
|
||||
|
||||
def write_summary(path: Path, status: str, category: str) -> None:
|
||||
finding = status.startswith("FAIL_")
|
||||
payload = {
|
||||
"issueId": "AUTO_BACKUP",
|
||||
"status": status,
|
||||
"conclusion": status,
|
||||
"findingCount": 1 if finding else 0,
|
||||
"findings": ([{"source": "offhost_deadman", "status": status, "severity": "P0", "detail": category}] if finding else []),
|
||||
"privacy": "status/count/category only; no credentials, host addresses, absolute paths, file names, or payloads",
|
||||
}
|
||||
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
fd, temporary = tempfile.mkstemp(prefix=".deadman-summary.", dir=path.parent)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, ensure_ascii=True, indent=2)
|
||||
handle.write("\n")
|
||||
os.chmod(temporary, 0o600)
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--archive-root", required=True, type=Path)
|
||||
parser.add_argument("--state-dir", required=True, type=Path)
|
||||
parser.add_argument("--webhook-file", required=True, type=Path)
|
||||
parser.add_argument("--alert-helper", required=True, type=Path)
|
||||
parser.add_argument("--max-age-seconds", type=int, default=93600)
|
||||
args = parser.parse_args()
|
||||
if args.max_age_seconds <= 0:
|
||||
fail("max age must be positive")
|
||||
try:
|
||||
args.state_dir.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
if args.state_dir.is_symlink() or not args.state_dir.is_dir():
|
||||
fail("state directory is invalid")
|
||||
os.chmod(args.state_dir, 0o700)
|
||||
status, category = validate_latest(args.archive_root, args.max_age_seconds)
|
||||
summary = args.state_dir / "last-deadman.json"
|
||||
write_summary(summary, status, category)
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(args.alert_helper), "--webhook-file", str(args.webhook_file), "--summary-file", str(summary), "--state-file", str(args.state_dir / "deadman-alert-state.json")],
|
||||
check=False,
|
||||
)
|
||||
except OSError:
|
||||
fail("dead-man local file operation failed")
|
||||
if result.returncode != 0:
|
||||
fail("alert evaluation failed")
|
||||
if status.startswith("FAIL_"):
|
||||
fail(category)
|
||||
print("[offhost-deadman][PASS] newest archive is fresh and valid")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -78,7 +78,7 @@ receiver() {
|
||||
"$ROOT_DIR/scripts/receive-offhost-backup.sh" "$@"
|
||||
}
|
||||
|
||||
CURRENT_RUN_ID="auto-$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
CURRENT_RUN_ID="auto-$(date -u +%Y%m%d)T000000Z"
|
||||
CURRENT_BATCH="hyb-oa-backup-$CURRENT_RUN_ID"
|
||||
|
||||
VALID_LAYOUT="$TEST_DIR/receiver-valid"
|
||||
@@ -92,15 +92,41 @@ pass "receiver finalizes one complete synthetic batch"
|
||||
receiver "$VALID_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$VALID_MANIFEST_SHA" >/dev/null
|
||||
pass "receiver treats an identical repeated finalize as idempotent"
|
||||
|
||||
SECOND_RUN_ID="auto-$(date -u +%Y%m%d)T000000Z"
|
||||
if [[ "$SECOND_RUN_ID" == "$CURRENT_RUN_ID" ]]; then
|
||||
SECOND_RUN_ID="auto-$(date -u +%Y%m%d)T000001Z"
|
||||
fi
|
||||
SECOND_BATCH="hyb-oa-backup-$SECOND_RUN_ID"
|
||||
make_batch "$VALID_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID"
|
||||
SECOND_MANIFEST_SHA="$(sha256sum "$VALID_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver rejects a different automatic batch for the same UTC day" \
|
||||
receiver "$VALID_LAYOUT" finalize-test "$SECOND_RUN_ID" "$SECOND_MANIFEST_SHA"
|
||||
receiver "$VALID_LAYOUT" finalize-test "$SECOND_RUN_ID" "$SECOND_MANIFEST_SHA" >/dev/null
|
||||
[[ ! -e "$VALID_LAYOUT/archive/$CURRENT_BATCH" && -d "$VALID_LAYOUT/archive/$SECOND_BATCH" ]]
|
||||
WEEKLY_BATCH="$(find "$VALID_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -print -quit)"
|
||||
[[ -n "$WEEKLY_BATCH" ]]
|
||||
[[ "$(stat -c %i "$WEEKLY_BATCH/database.dump")" == "$(stat -c %i "$VALID_LAYOUT/archive/$SECOND_BATCH/database.dump")" ]]
|
||||
pass "receiver atomically replaces a different batch for the same UTC day and rebuilds its weekly checkpoint"
|
||||
|
||||
THIRD_RUN_ID="auto-$(date -u +%Y%m%d)T000002Z"
|
||||
THIRD_BATCH="hyb-oa-backup-$THIRD_RUN_ID"
|
||||
make_batch "$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID"
|
||||
printf 'corrupt candidate\n' >>"$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial/database.dump"
|
||||
THIRD_MANIFEST_SHA="$(sha256sum "$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver validates a same-day replacement before touching the current archive" \
|
||||
receiver "$VALID_LAYOUT" finalize-test "$THIRD_RUN_ID" "$THIRD_MANIFEST_SHA"
|
||||
[[ -d "$VALID_LAYOUT/archive/$SECOND_BATCH" && ! -e "$VALID_LAYOUT/archive/$THIRD_BATCH" ]]
|
||||
pass "receiver preserves the current daily archive when a replacement candidate is corrupt"
|
||||
|
||||
ROLLBACK_LAYOUT="$TEST_DIR/receiver-replacement-rollback"
|
||||
make_layout "$ROLLBACK_LAYOUT"
|
||||
make_batch "$ROLLBACK_LAYOUT/inbox/${CURRENT_BATCH}.partial" "$CURRENT_RUN_ID"
|
||||
ROLLBACK_FIRST_SHA="$(sha256sum "$ROLLBACK_LAYOUT/inbox/${CURRENT_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
receiver "$ROLLBACK_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$ROLLBACK_FIRST_SHA" >/dev/null
|
||||
ROLLBACK_WEEKLY="$(find "$ROLLBACK_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -print -quit)"
|
||||
find "$ROLLBACK_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -exec rm -rf -- {} +
|
||||
printf 'invalid weekly target\n' >"$ROLLBACK_WEEKLY"
|
||||
make_batch "$ROLLBACK_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID"
|
||||
ROLLBACK_SECOND_SHA="$(sha256sum "$ROLLBACK_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver rolls back the daily archive when weekly checkpoint publication fails" \
|
||||
receiver "$ROLLBACK_LAYOUT" finalize-test "$SECOND_RUN_ID" "$ROLLBACK_SECOND_SHA"
|
||||
[[ -d "$ROLLBACK_LAYOUT/archive/$CURRENT_BATCH" && ! -e "$ROLLBACK_LAYOUT/archive/$SECOND_BATCH" ]]
|
||||
pass "receiver restores the prior daily point after a later replacement step fails"
|
||||
|
||||
negative_receiver_case() {
|
||||
local label="$1"
|
||||
@@ -225,7 +251,7 @@ if [[ "${1:-}" == "compose" ]]; then
|
||||
args=" $* "
|
||||
if [[ "$args" == *" ps -q server "* ]]; then echo server-id; exit 0; fi
|
||||
if [[ "$args" == *" ps -q db "* ]]; then echo db-id; exit 0; fi
|
||||
if [[ "$args" == *" exec -T server "* ]]; then cat >>"$MOCK_ALERT_LOG"; exit 0; fi
|
||||
if [[ "$args" == *" exec -T server "* ]]; then exit 99; fi
|
||||
fi
|
||||
if [[ "${1:-}" == "inspect" ]]; then
|
||||
template="${3:-}"
|
||||
@@ -260,6 +286,7 @@ MOCK
|
||||
|
||||
cat >"$MOCK_BIN/rsync" <<'MOCK'
|
||||
#!/usr/bin/env bash
|
||||
echo 'synthetic transfer diagnostic' >&2
|
||||
exit "${MOCK_RSYNC_EXIT:-1}"
|
||||
MOCK
|
||||
cat >"$MOCK_BIN/ssh" <<'MOCK'
|
||||
@@ -267,6 +294,34 @@ cat >"$MOCK_BIN/ssh" <<'MOCK'
|
||||
exit "${MOCK_SSH_EXIT:-0}"
|
||||
MOCK
|
||||
chmod +x "$MOCK_BIN/docker" "$MOCK_BIN/rsync" "$MOCK_BIN/ssh"
|
||||
WECOM_TEST_HOST='qyapi.weixin.qq.com'
|
||||
WECOM_TEST_PATH='/cgi-bin/webhook/send'
|
||||
printf 'https://%s%s?key=%s\n' "$WECOM_TEST_HOST" "$WECOM_TEST_PATH" \
|
||||
'00000000-0000-0000-0000-000000000000' >"$TEST_DIR/wecom-webhook"
|
||||
chmod 0600 "$TEST_DIR/wecom-webhook"
|
||||
ALERT_CAPTURE="$TEST_DIR/alert-capture.jsonl"
|
||||
cat >"$TEST_DIR/failure-summary.json" <<'JSON'
|
||||
{"issueId":"AUTO_BACKUP","status":"FAIL_AUTOMATIC_BACKUP","conclusion":"FAIL_AUTOMATIC_BACKUP","findingCount":1,"findings":[{"source":"automatic_backup","status":"FAIL_AUTOMATIC_BACKUP","severity":"P0","detail":"synthetic_failure"}]}
|
||||
JSON
|
||||
chmod 0600 "$TEST_DIR/failure-summary.json"
|
||||
chmod 0644 "$TEST_DIR/wecom-webhook"
|
||||
expect_failure "group alert helper rejects a webhook file that is not mode 0600" env \
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \
|
||||
python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook" \
|
||||
--summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json"
|
||||
chmod 0600 "$TEST_DIR/wecom-webhook"
|
||||
ln -s "$TEST_DIR/wecom-webhook" "$TEST_DIR/wecom-webhook-link"
|
||||
expect_failure "group alert helper rejects a symbolic-link webhook file" env \
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \
|
||||
python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook-link" \
|
||||
--summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json"
|
||||
printf 'https://%s%s?key=%s&extra=forbidden\n' "$WECOM_TEST_HOST" "$WECOM_TEST_PATH" \
|
||||
'00000000-0000-0000-0000-000000000000' >"$TEST_DIR/wecom-webhook-invalid"
|
||||
chmod 0600 "$TEST_DIR/wecom-webhook-invalid"
|
||||
expect_failure "group alert helper rejects a webhook URL with an unexpected query field" env \
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \
|
||||
python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook-invalid" \
|
||||
--summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json"
|
||||
|
||||
wrapper_env=(
|
||||
env PATH="$MOCK_BIN:$PATH" REMOTE_DIR="$WRAPPER_ROOT"
|
||||
@@ -275,7 +330,9 @@ wrapper_env=(
|
||||
STORAGE_HOST_DIR="$WRAPPER_STORAGE" BUILD_SHA="$BUILD_SHA" VITE_BUILD_SHA="$VITE_BUILD_SHA"
|
||||
AUTOMATIC_BACKUP_DIR="$WRAPPER_BACKUPS" AUTOMATIC_BACKUP_STATE_DIR="$WRAPPER_STATE"
|
||||
OFFHOST_TARGET="hybbackup@backup-alias" OFFHOST_SSH_KEY="$TEST_DIR/offhost-key"
|
||||
OFFHOST_KNOWN_HOSTS="$TEST_DIR/known-hosts" ALERT_RECIPIENT_EMPLOYEE_NO="OPS001"
|
||||
OFFHOST_KNOWN_HOSTS="$TEST_DIR/known-hosts" ALERT_WEBHOOK_FILE="$TEST_DIR/wecom-webhook"
|
||||
ALERT_HELPER="$ROOT_DIR/scripts/send-wecom-group-alert.py"
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE"
|
||||
)
|
||||
|
||||
exec 8>"$WRAPPER_STATE/automatic-backup.lock"
|
||||
@@ -305,6 +362,9 @@ assert payload["status"] == "FAIL_AUTOMATIC_BACKUP"
|
||||
assert not any(value in serialized for value in ("/tmp/", "postgresql://", "backup-alias", "fixture"))
|
||||
PYTHON
|
||||
pass "wrapper failure summary is category-only and redacted"
|
||||
[[ -f "$WRAPPER_STATE/last-error.log" && "$(stat -c %a "$WRAPPER_STATE/last-error.log")" == "600" ]]
|
||||
grep -Fq 'synthetic transfer diagnostic' "$WRAPPER_STATE/last-error.log"
|
||||
pass "wrapper retains the latest command diagnostic locally with mode 0600"
|
||||
|
||||
find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' -exec rm -rf -- {} +
|
||||
for day in 01 02 03 04 05 06 07; do
|
||||
@@ -315,6 +375,8 @@ done
|
||||
[[ -f "$WRAPPER_STATE/last-success.json" ]]
|
||||
[[ "$(find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' | wc -l)" == "7" ]]
|
||||
pass "off-host finalize success writes state and then prunes source retention to seven"
|
||||
[[ ! -e "$WRAPPER_STATE/last-error.log" ]]
|
||||
pass "wrapper clears the retained diagnostic after a successful off-host run"
|
||||
|
||||
rm -f "$WRAPPER_STATE/last-success.json"
|
||||
expect_failure "stale watchdog rejects a missing success record" \
|
||||
@@ -329,7 +391,7 @@ with open(sys.argv[1], "w", encoding="utf-8") as handle:
|
||||
"schemaVersion": 1,
|
||||
"status": "complete",
|
||||
"completedAt": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
"runId": "auto-synthetic",
|
||||
"runId": "auto-20260727T000000Z",
|
||||
"manifestSha256": "0" * 64,
|
||||
"composeProject": "hyb-oa-current",
|
||||
"buildSha": sys.argv[2],
|
||||
@@ -345,12 +407,89 @@ payload["buildSha"] = "0" * 40
|
||||
with open(sys.argv[1], "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle)
|
||||
PYTHON
|
||||
expect_failure "stale watchdog rejects a success record from another build" \
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale >/dev/null
|
||||
python3 - "$WRAPPER_STATE/last-run.json" <<'PYTHON'
|
||||
import json
|
||||
import sys
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
assert payload["status"] == "PASS_WITH_FINDINGS"
|
||||
assert payload["findings"][0]["detail"] == "recovery_point_build_differs"
|
||||
PYTHON
|
||||
pass "stale watchdog downgrades a fresh recovery point from another build to one finding"
|
||||
mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json"
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale >/dev/null
|
||||
pass "stale watchdog accepts a fresh project/build-bound success record"
|
||||
|
||||
cp "$WRAPPER_STATE/last-success.json" "$WRAPPER_STATE/last-success.valid.json"
|
||||
python3 - "$WRAPPER_STATE/last-success.json" <<'PYTHON'
|
||||
import json
|
||||
import sys
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
payload["composeProject"] = "different-project"
|
||||
with open(sys.argv[1], "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle)
|
||||
PYTHON
|
||||
expect_failure "stale watchdog hard-fails a fresh record from another Compose project" \
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
|
||||
mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json"
|
||||
cp "$WRAPPER_STATE/last-success.json" "$WRAPPER_STATE/last-success.valid.json"
|
||||
python3 - "$WRAPPER_STATE/last-success.json" <<'PYTHON'
|
||||
import json
|
||||
import sys
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
payload["manifestSha256"] = "invalid"
|
||||
with open(sys.argv[1], "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle)
|
||||
PYTHON
|
||||
expect_failure "stale watchdog hard-fails a malformed success record" \
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
|
||||
mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json"
|
||||
|
||||
DEADMAN_LAYOUT="$TEST_DIR/deadman"
|
||||
mkdir -p "$DEADMAN_LAYOUT/archive" "$DEADMAN_LAYOUT/state"
|
||||
cp -a "$VALID_LAYOUT/archive/$SECOND_BATCH" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH"
|
||||
DEADMAN_CAPTURE="$TEST_DIR/deadman-alerts.jsonl"
|
||||
deadman_env=(env HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$DEADMAN_CAPTURE")
|
||||
"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null
|
||||
[[ ! -e "$DEADMAN_CAPTURE" ]]
|
||||
pass "ARM dead-man accepts a fresh verified archive without sending a healthy baseline"
|
||||
DEADMAN_CORRUPT_STATE="$TEST_DIR/deadman-corrupt-state"
|
||||
DEADMAN_CORRUPT_CAPTURE="$TEST_DIR/deadman-corrupt-alerts.jsonl"
|
||||
mkdir "$DEADMAN_CORRUPT_STATE"
|
||||
chmod 0600 "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
|
||||
printf 'corruption\n' >>"$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
|
||||
expect_failure "ARM dead-man rejects a corrupt newest finalized archive" env \
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$DEADMAN_CORRUPT_CAPTURE" \
|
||||
python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_CORRUPT_STATE" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py"
|
||||
cp "$VALID_LAYOUT/archive/$SECOND_BATCH/database.dump" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
|
||||
chmod 0400 "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
|
||||
pass "ARM dead-man detects archive corruption independently of source state"
|
||||
mv "$DEADMAN_LAYOUT/archive/$SECOND_BATCH" "$TEST_DIR/deadman-withheld-batch"
|
||||
expect_failure "ARM dead-man detects that the source no longer produces a finalized archive" \
|
||||
"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py"
|
||||
[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "1" ]]
|
||||
expect_failure "ARM dead-man suppresses an identical repeated failure alert" \
|
||||
"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py"
|
||||
[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "1" ]]
|
||||
pass "ARM dead-man keeps repeated identical failure alerts deduplicated"
|
||||
mv "$TEST_DIR/deadman-withheld-batch" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH"
|
||||
"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null
|
||||
[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "2" ]]
|
||||
pass "ARM dead-man sends one recovery notification after finalized archives resume"
|
||||
|
||||
DRIFT_BUILD_SHA="0123456789abcdef0123456789abcdef01234567"
|
||||
expect_failure "wrapper rejects an active runtime build that drifted from its installed contract" \
|
||||
"${wrapper_env[@]}" BUILD_SHA="$DRIFT_BUILD_SHA" VITE_BUILD_SHA="${DRIFT_BUILD_SHA:0:12}" \
|
||||
|
||||
@@ -60,6 +60,7 @@ validate_layout() {
|
||||
[[ "$DAILY_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "DAILY_KEEP must be positive"
|
||||
[[ "$WEEKLY_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "WEEKLY_KEEP must be positive"
|
||||
command -v "$PYTHON_BIN" >/dev/null 2>&1 || fail "python3 is required"
|
||||
command -v flock >/dev/null 2>&1 || fail "flock is required"
|
||||
}
|
||||
|
||||
validate_run_id() {
|
||||
@@ -155,11 +156,44 @@ prune_named_directories() {
|
||||
done
|
||||
}
|
||||
|
||||
publish_weekly_checkpoint() {
|
||||
local archive_dir="$1"
|
||||
local weekly_dir="$2"
|
||||
local replace_existing="$3"
|
||||
if [[ -e "$weekly_dir" && "$replace_existing" != "1" ]]; then
|
||||
return 0
|
||||
fi
|
||||
local weekly_temp="${weekly_dir}.new.$$"
|
||||
local weekly_rollback="${weekly_dir}.rollback.$$"
|
||||
[[ ! -e "$weekly_temp" && ! -e "$weekly_rollback" ]] || return 1
|
||||
if ! cp -al -- "$archive_dir" "$weekly_temp"; then
|
||||
rm -rf -- "$weekly_temp"
|
||||
return 1
|
||||
fi
|
||||
if [[ -e "$weekly_dir" ]]; then
|
||||
[[ -d "$weekly_dir" && ! -L "$weekly_dir" ]] || return 1
|
||||
mv -- "$weekly_dir" "$weekly_rollback" || return 1
|
||||
fi
|
||||
if ! mv -- "$weekly_temp" "$weekly_dir"; then
|
||||
rm -rf -- "$weekly_temp"
|
||||
if [[ -e "$weekly_rollback" ]]; then
|
||||
mv -- "$weekly_rollback" "$weekly_dir"
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
if [[ -e "$weekly_rollback" ]]; then
|
||||
rm -rf -- "$weekly_rollback"
|
||||
fi
|
||||
}
|
||||
|
||||
finalize_batch() {
|
||||
local run_id="$1"
|
||||
local expected_manifest_sha="$2"
|
||||
validate_run_id "$run_id"
|
||||
validate_layout
|
||||
exec 9>"$BACKUP_ARCHIVE_DIR/.finalize.lock"
|
||||
chmod 0600 "$BACKUP_ARCHIVE_DIR/.finalize.lock"
|
||||
flock -n 9 || fail "another finalize operation is already running"
|
||||
|
||||
local batch_name="hyb-oa-backup-$run_id"
|
||||
local partial_dir="$BACKUP_INBOX_DIR/${batch_name}.partial"
|
||||
@@ -173,15 +207,32 @@ finalize_batch() {
|
||||
|
||||
local run_day="${run_id#auto-}"
|
||||
run_day="${run_day%%T*}"
|
||||
local existing_for_day
|
||||
existing_for_day="$(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 -type d \
|
||||
-name "hyb-oa-backup-auto-${run_day}T*Z" ! -name "$batch_name" -print -quit)"
|
||||
[[ -z "$existing_for_day" ]] || fail "another automatic archive already exists for this UTC day"
|
||||
local existing_for_day="" replacement_rollback=""
|
||||
local -a existing_for_day_entries=()
|
||||
mapfile -t existing_for_day_entries < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 -type d \
|
||||
-name "hyb-oa-backup-auto-${run_day}T*Z" ! -name "$batch_name" -print)
|
||||
[[ "${#existing_for_day_entries[@]}" -le 1 ]] || fail "daily archive invariant is already violated"
|
||||
if [[ "${#existing_for_day_entries[@]}" -eq 1 ]]; then
|
||||
existing_for_day="${existing_for_day_entries[0]}"
|
||||
fi
|
||||
|
||||
if [[ "$already_archived" == "0" ]]; then
|
||||
validate_batch "$partial_dir" "$run_id" "$expected_manifest_sha"
|
||||
chmod 0400 "$partial_dir/database.dump" "$partial_dir/storage.tar.gz" "$partial_dir/manifest.json"
|
||||
mv -- "$partial_dir" "$archive_dir"
|
||||
if [[ -n "$existing_for_day" ]]; then
|
||||
[[ "$existing_for_day" =~ /hyb-oa-backup-auto-${run_day}T[0-9]{6}Z$ \
|
||||
&& -d "$existing_for_day" && ! -L "$existing_for_day" ]] \
|
||||
|| fail "existing daily archive is invalid"
|
||||
replacement_rollback="$BACKUP_ARCHIVE_DIR/.daily-replacement-${run_day}.$$"
|
||||
[[ ! -e "$replacement_rollback" ]] || fail "daily replacement rollback target already exists"
|
||||
mv -- "$existing_for_day" "$replacement_rollback"
|
||||
fi
|
||||
if ! mv -- "$partial_dir" "$archive_dir"; then
|
||||
if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then
|
||||
mv -- "$replacement_rollback" "$existing_for_day"
|
||||
fi
|
||||
fail "daily archive publication failed"
|
||||
fi
|
||||
fi
|
||||
|
||||
local iso_week
|
||||
@@ -194,8 +245,15 @@ print(f"week-{iso.year:04d}-W{iso.week:02d}")
|
||||
PYTHON
|
||||
)"
|
||||
local weekly_dir="$BACKUP_WEEKLY_DIR/$iso_week"
|
||||
if [[ ! -e "$weekly_dir" ]]; then
|
||||
cp -al -- "$archive_dir" "$weekly_dir"
|
||||
if ! publish_weekly_checkpoint "$archive_dir" "$weekly_dir" "$([[ -n "$replacement_rollback" ]] && echo 1 || echo 0)"; then
|
||||
if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then
|
||||
rm -rf -- "$archive_dir"
|
||||
mv -- "$replacement_rollback" "$existing_for_day"
|
||||
fi
|
||||
fail "weekly checkpoint update failed"
|
||||
fi
|
||||
if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then
|
||||
rm -rf -- "$replacement_rollback"
|
||||
fi
|
||||
|
||||
prune_named_directories "$BACKUP_ARCHIVE_DIR" "hyb-oa-backup-auto-" "$DAILY_KEEP"
|
||||
|
||||
+55
-16
@@ -15,9 +15,10 @@ AUTOMATIC_BACKUP_STATE_DIR="${AUTOMATIC_BACKUP_STATE_DIR:-}"
|
||||
OFFHOST_TARGET="${OFFHOST_TARGET:-}"
|
||||
OFFHOST_SSH_KEY="${OFFHOST_SSH_KEY:-}"
|
||||
OFFHOST_KNOWN_HOSTS="${OFFHOST_KNOWN_HOSTS:-}"
|
||||
ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
|
||||
ALERT_HELPER="${ALERT_HELPER:-$ROOT_DIR/scripts/send-wecom-group-alert.py}"
|
||||
AUTOMATIC_BACKUP_KEEP="${AUTOMATIC_BACKUP_KEEP:-7}"
|
||||
AUTOMATIC_BACKUP_MAX_AGE_SECONDS="${AUTOMATIC_BACKUP_MAX_AGE_SECONDS:-86400}"
|
||||
ALERT_RECIPIENT_EMPLOYEE_NO="${ALERT_RECIPIENT_EMPLOYEE_NO:-HYD000006}"
|
||||
MODE="${1:-run}"
|
||||
|
||||
CURRENT_PHASE="preflight"
|
||||
@@ -79,7 +80,12 @@ validate_common() {
|
||||
|| fail "frontend build SHA must match the full build prefix"
|
||||
[[ "$AUTOMATIC_BACKUP_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "backup retention must be positive"
|
||||
[[ "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" =~ ^[1-9][0-9]*$ ]] || fail "stale threshold must be positive"
|
||||
[[ "$ALERT_RECIPIENT_EMPLOYEE_NO" =~ ^[A-Za-z0-9_-]{1,80}$ ]] || fail "alert recipient id is invalid"
|
||||
validate_absolute_path ALERT_WEBHOOK_FILE "$ALERT_WEBHOOK_FILE"
|
||||
validate_absolute_path ALERT_HELPER "$ALERT_HELPER"
|
||||
[[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
|
||||
&& "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" ]] \
|
||||
|| fail "alert webhook file must exist with mode 0600"
|
||||
[[ -f "$ALERT_HELPER" && ! -L "$ALERT_HELPER" ]] || fail "alert helper is missing"
|
||||
if [[ "$AUTOMATIC_BACKUP_DIR" == "$AUTOMATIC_BACKUP_STATE_DIR" \
|
||||
|| "$AUTOMATIC_BACKUP_DIR" == "$AUTOMATIC_BACKUP_STATE_DIR"/* \
|
||||
|| "$AUTOMATIC_BACKUP_STATE_DIR" == "$AUTOMATIC_BACKUP_DIR"/* ]]; then
|
||||
@@ -218,10 +224,25 @@ PYTHON
|
||||
send_summary_alert() {
|
||||
local state_name="$1"
|
||||
[[ -n "$SUMMARY_FILE" && -f "$SUMMARY_FILE" ]] || return 1
|
||||
[[ -n "$SERVER_CONTAINER" ]] || resolve_active_runtime
|
||||
compose exec -T server sh -lc \
|
||||
"cd /app/server && WARCH3_OPS_ALERT_SEND=1 npm run send:wecom-ops-alert -- --summary-file=- --recipient-employee-no=${ALERT_RECIPIENT_EMPLOYEE_NO} --send --state-file=/app/artifacts/qa-flow/${state_name} --format=json" \
|
||||
<"$SUMMARY_FILE" >/dev/null
|
||||
python3 "$ALERT_HELPER" \
|
||||
--webhook-file "$ALERT_WEBHOOK_FILE" \
|
||||
--summary-file "$SUMMARY_FILE" \
|
||||
--state-file "$AUTOMATIC_BACKUP_STATE_DIR/$state_name" >/dev/null
|
||||
}
|
||||
|
||||
capture_command() {
|
||||
local phase="$1"
|
||||
shift
|
||||
local temporary="$AUTOMATIC_BACKUP_STATE_DIR/.${phase}.diagnostic.$$"
|
||||
: >"$temporary"
|
||||
chmod 0600 "$temporary"
|
||||
if "$@" >"$temporary" 2>&1; then
|
||||
rm -f -- "$temporary"
|
||||
return 0
|
||||
fi
|
||||
mv -f -- "$temporary" "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log"
|
||||
chmod 0600 "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log"
|
||||
return 1
|
||||
}
|
||||
|
||||
on_error() {
|
||||
@@ -320,11 +341,12 @@ print(f"{unquote(value.username)}\t{database_name}")
|
||||
[[ "$source_image_digest" =~ ^sha256:[0-9a-f]{64}$ ]] || fail "active server image digest is invalid"
|
||||
|
||||
CURRENT_PHASE="paired_backup"
|
||||
if ! DATABASE_URL="$database_url" DB_TOOL_MODE=docker DB_CONTAINER="$DB_CONTAINER" \
|
||||
if ! capture_command paired_backup env \
|
||||
DATABASE_URL="$database_url" DB_TOOL_MODE=docker DB_CONTAINER="$DB_CONTAINER" \
|
||||
POSTGRES_DB="$postgres_db" POSTGRES_USER="$postgres_user" POSTGRES_PASSWORD="$postgres_password" \
|
||||
STORAGE_DIR_HOST="$STORAGE_HOST_DIR" BACKUP_DIR="$AUTOMATIC_BACKUP_DIR" \
|
||||
BACKUP_RUN_ID="$run_id" BUILD_SHA="$BUILD_SHA" SOURCE_IMAGE_DIGEST="$source_image_digest" \
|
||||
"$ROOT_DIR/scripts/backup-all.sh" >/dev/null 2>&1; then
|
||||
"$ROOT_DIR/scripts/backup-all.sh"; then
|
||||
fail "paired backup command failed"
|
||||
fi
|
||||
[[ -d "$batch_dir" && -f "$batch_dir/manifest.json" ]] || fail "complete local batch was not published"
|
||||
@@ -343,14 +365,14 @@ print(f"{unquote(value.username)}\t{database_name}")
|
||||
"$OFFHOST_SSH_KEY" "$OFFHOST_KNOWN_HOSTS"
|
||||
|
||||
CURRENT_PHASE="offhost_transfer"
|
||||
if ! rsync -a --partial --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= -e "$rsync_shell" \
|
||||
"$batch_dir/" "$OFFHOST_TARGET:${batch_name}.partial/" >/dev/null 2>&1; then
|
||||
if ! capture_command offhost_transfer rsync -a --partial --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= -e "$rsync_shell" \
|
||||
"$batch_dir/" "$OFFHOST_TARGET:${batch_name}.partial/"; then
|
||||
fail "off-host transfer failed"
|
||||
fi
|
||||
|
||||
CURRENT_PHASE="offhost_finalize"
|
||||
if ! ssh "${ssh_options[@]}" "$OFFHOST_TARGET" \
|
||||
"hyb-backup-finalize $run_id $manifest_sha" >/dev/null 2>&1; then
|
||||
if ! capture_command offhost_finalize ssh "${ssh_options[@]}" "$OFFHOST_TARGET" \
|
||||
"hyb-backup-finalize $run_id $manifest_sha"; then
|
||||
fail "off-host finalize failed"
|
||||
fi
|
||||
|
||||
@@ -359,6 +381,7 @@ print(f"{unquote(value.username)}\t{database_name}")
|
||||
prune_local_backups "$batch_name"
|
||||
write_summary "PASS_AUTOMATIC_BACKUP_COMPLETE" 0 "offhost_verified"
|
||||
send_summary_alert "automatic-backup-alert-state.json" || true
|
||||
rm -f -- "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log"
|
||||
trap - ERR
|
||||
echo "[automatic-backup][PASS] paired batch finalized off-host"
|
||||
}
|
||||
@@ -369,9 +392,11 @@ check_stale() {
|
||||
CURRENT_PHASE="stale_check"
|
||||
resolve_active_runtime
|
||||
local success_file="$AUTOMATIC_BACKUP_STATE_DIR/last-success.json"
|
||||
if ! python3 - "$success_file" "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" "$COMPOSE_PROJECT_NAME" "$BUILD_SHA" <<'PYTHON'
|
||||
local stale_result
|
||||
if ! stale_result="$(python3 - "$success_file" "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" "$COMPOSE_PROJECT_NAME" "$BUILD_SHA" <<'PYTHON'
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
@@ -382,15 +407,29 @@ with open(file_path, "r", encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
if payload.get("schemaVersion") != 1 or payload.get("status") != "complete":
|
||||
raise SystemExit(1)
|
||||
if payload.get("composeProject") != expected_project or payload.get("buildSha") != expected_build:
|
||||
if payload.get("composeProject") != expected_project:
|
||||
raise SystemExit(1)
|
||||
if not re.fullmatch(r"auto-[0-9]{8}T[0-9]{6}Z", payload.get("runId", "")):
|
||||
raise SystemExit(1)
|
||||
if not re.fullmatch(r"[0-9a-f]{64}", payload.get("manifestSha256", "")):
|
||||
raise SystemExit(1)
|
||||
if not re.fullmatch(r"[0-9a-fA-F]{40,64}", payload.get("buildSha", "")):
|
||||
raise SystemExit(1)
|
||||
completed = datetime.fromisoformat(payload["completedAt"].replace("Z", "+00:00"))
|
||||
age = (datetime.now(timezone.utc) - completed).total_seconds()
|
||||
if age < 0 or age > int(max_age):
|
||||
raise SystemExit(1)
|
||||
print("BUILD_MISMATCH" if payload.get("buildSha") != expected_build else "FRESH")
|
||||
PYTHON
|
||||
then
|
||||
fail "latest off-host success is missing, invalid, mismatched, or stale"
|
||||
)"; then
|
||||
fail "latest off-host success is missing, invalid, project-mismatched, or stale"
|
||||
fi
|
||||
if [[ "$stale_result" == "BUILD_MISMATCH" ]]; then
|
||||
write_summary "PASS_WITH_FINDINGS" 1 "recovery_point_build_differs"
|
||||
send_summary_alert "automatic-backup-alert-state.json" || true
|
||||
trap - ERR
|
||||
echo "[automatic-backup][PASS_WITH_FINDINGS] off-host recovery point is fresh but build differs"
|
||||
return 0
|
||||
fi
|
||||
write_summary "PASS_AUTOMATIC_BACKUP_FRESH" 0 "fresh_offhost_recovery_point"
|
||||
send_summary_alert "automatic-backup-alert-state.json" || true
|
||||
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Send a redacted, deduplicated OA operations alert to a WeCom group robot."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SAFE_TOKEN = re.compile(r"^[A-Za-z0-9_-]{1,160}$")
|
||||
ALLOWED_STATUSES = {
|
||||
"FAIL_AUTOMATIC_BACKUP",
|
||||
"FAIL_OFFHOST_DEADMAN",
|
||||
"PASS_AUTOMATIC_BACKUP_COMPLETE",
|
||||
"PASS_AUTOMATIC_BACKUP_FRESH",
|
||||
"PASS_OFFHOST_DEADMAN_FRESH",
|
||||
"PASS_WITH_FINDINGS",
|
||||
}
|
||||
|
||||
|
||||
def fail(message: str) -> "NoReturn":
|
||||
print(f"[wecom-group-alert][ERROR] {message}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def validate_private_file(path: Path, label: str) -> None:
|
||||
try:
|
||||
info = path.lstat()
|
||||
except FileNotFoundError:
|
||||
fail(f"{label} is missing")
|
||||
if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
|
||||
fail(f"{label} must be a regular non-symbolic file")
|
||||
if stat.S_IMODE(info.st_mode) != 0o600:
|
||||
fail(f"{label} mode must be 0600")
|
||||
if info.st_uid != os.geteuid():
|
||||
fail(f"{label} must be owned by the executing account")
|
||||
|
||||
|
||||
def read_webhook(path: Path) -> str:
|
||||
validate_private_file(path, "webhook file")
|
||||
try:
|
||||
value = path.read_text(encoding="utf-8").strip()
|
||||
except OSError:
|
||||
fail("webhook file cannot be read")
|
||||
if "\n" in value or "\r" in value:
|
||||
fail("webhook file must contain one line")
|
||||
parsed = urllib.parse.urlparse(value)
|
||||
query = urllib.parse.parse_qs(parsed.query, strict_parsing=True)
|
||||
if (
|
||||
parsed.scheme != "https"
|
||||
or parsed.hostname != "qyapi.weixin.qq.com"
|
||||
or parsed.netloc != "qyapi.weixin.qq.com"
|
||||
or parsed.path != "/cgi-bin/webhook/send"
|
||||
or set(query) != {"key"}
|
||||
or len(query["key"]) != 1
|
||||
or not re.fullmatch(r"[A-Za-z0-9-]{16,128}", query["key"][0])
|
||||
or parsed.fragment
|
||||
):
|
||||
fail("webhook file format is invalid")
|
||||
return value
|
||||
|
||||
|
||||
def load_summary(path: Path) -> tuple[str, str, int]:
|
||||
validate_private_file(path, "summary file")
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
fail("summary file is invalid")
|
||||
status = payload.get("status")
|
||||
findings = payload.get("findings") or []
|
||||
finding_count = payload.get("findingCount")
|
||||
if status not in ALLOWED_STATUSES or not isinstance(finding_count, int):
|
||||
fail("summary status is invalid")
|
||||
if finding_count != len(findings) or finding_count not in {0, 1}:
|
||||
fail("summary finding count is invalid")
|
||||
category = "healthy"
|
||||
if findings:
|
||||
category = findings[0].get("detail", "")
|
||||
if not isinstance(category, str) or not SAFE_TOKEN.fullmatch(category):
|
||||
fail("summary category is invalid")
|
||||
return status, category, finding_count
|
||||
|
||||
|
||||
def load_previous(path: Path) -> dict[str, object]:
|
||||
if not path.exists():
|
||||
return {}
|
||||
validate_private_file(path, "alert state file")
|
||||
try:
|
||||
value = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
fail("alert state file is invalid")
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def atomic_write(path: Path, payload: dict[str, object]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
os.chmod(path.parent, 0o700)
|
||||
fd, temp_name = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
|
||||
try:
|
||||
os.fchmod(fd, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, ensure_ascii=True, indent=2)
|
||||
handle.write("\n")
|
||||
os.replace(temp_name, path)
|
||||
except BaseException:
|
||||
try:
|
||||
os.unlink(temp_name)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def should_send(previous: dict[str, object], status: str, category: str) -> bool:
|
||||
previous_status = previous.get("status")
|
||||
previous_category = previous.get("category")
|
||||
if status.startswith("FAIL_") or status == "PASS_WITH_FINDINGS":
|
||||
return previous_status != status or previous_category != category
|
||||
return isinstance(previous_status, str) and (
|
||||
previous_status.startswith("FAIL_") or previous_status == "PASS_WITH_FINDINGS"
|
||||
)
|
||||
|
||||
|
||||
def safe_text(status: str, category: str) -> str:
|
||||
if status.startswith("FAIL_"):
|
||||
headline = "内部 OA 运维告警"
|
||||
elif status == "PASS_WITH_FINDINGS":
|
||||
headline = "内部 OA 运维提醒"
|
||||
else:
|
||||
headline = "内部 OA 运维恢复通知"
|
||||
return f"{headline}\n状态：{status}\n类别：{category}\n请按恢复卡检查，不要在群内粘贴原始日志。"
|
||||
|
||||
|
||||
def send(webhook: str, text: str) -> None:
|
||||
body = json.dumps({"msgtype": "text", "text": {"content": text}}).encode("utf-8")
|
||||
capture = os.environ.get("HYB_WECOM_ALERT_TEST_CAPTURE")
|
||||
if capture:
|
||||
if os.environ.get("HYB_WECOM_ALERT_TEST_MODE") != "1":
|
||||
fail("test capture is disabled")
|
||||
capture_path = Path(capture)
|
||||
with capture_path.open("ab") as handle:
|
||||
handle.write(body + b"\n")
|
||||
os.chmod(capture_path, 0o600)
|
||||
return
|
||||
request = urllib.request.Request(
|
||||
webhook,
|
||||
data=body,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
result = json.loads(response.read().decode("utf-8"))
|
||||
except Exception:
|
||||
fail("group robot request failed")
|
||||
if result.get("errcode") != 0:
|
||||
fail("group robot rejected the alert")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--webhook-file", required=True, type=Path)
|
||||
parser.add_argument("--summary-file", required=True, type=Path)
|
||||
parser.add_argument("--state-file", required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
status, category, _ = load_summary(args.summary_file)
|
||||
previous = load_previous(args.state_file)
|
||||
send_now = should_send(previous, status, category)
|
||||
if send_now:
|
||||
webhook = read_webhook(args.webhook_file)
|
||||
send(webhook, safe_text(status, category))
|
||||
atomic_write(args.state_file, {"schemaVersion": 1, "status": status, "category": category})
|
||||
except OSError:
|
||||
fail("local alert file operation failed")
|
||||
print(f"[wecom-group-alert][PASS] evaluated sent={str(send_now).lower()}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
在新工单中引用
屏蔽一个用户