diff --git a/.trellis/spec/hyb-oa/deployment-trialrun.md b/.trellis/spec/hyb-oa/deployment-trialrun.md index 3a6f5416..55c28a3c 100644 --- a/.trellis/spec/hyb-oa/deployment-trialrun.md +++ b/.trellis/spec/hyb-oa/deployment-trialrun.md @@ -952,6 +952,12 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ destination match. Do not split human-formatted Go-template output on tabs or spaces: `println` inserts argument spacing that can make a visually correct line fail exact path comparison on a real daemon. +- Database connection values used by the automatic wrapper must be exported in + an isolated child shell before `capture_command`; do not invoke `env` with a + complete `DATABASE_URL` or password assignment in its argv. Docker backup and + shadow-restore commands inherit `PGPASSWORD` with + `PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD ...`; never use a + value-bearing `-e PGPASSWORD=...` Docker argument. - Transfer only a complete `auto-` three-file batch to ARM inbox. The forced key allows write-only `rrsync` or an exact finalize command; it rejects shell, sender/read and destructive rsync options. @@ -962,6 +968,22 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ point to a same-filesystem rollback name, publish the candidate, rebuild that ISO week's hard-link checkpoint, then remove the rollback. Any publication or weekly-checkpoint failure restores the prior daily point. +- After taking the finalize lock and before reading the incoming partial, ARM + reconciles interrupted same-filesystem transactions. A valid + `.daily-replacement-.*` is deleted only when one valid canonical + archive for that day exists; when the canonical is absent it is renamed back + from its own verified manifest identity. Invalid, linked, duplicate or + otherwise ambiguous daily state fails closed and preserves every object. +- Weekly `.rollback.*` is authoritative only after full batch validation and is + restored when the canonical checkpoint is absent. Weekly `.new.*` is never an + authoritative source; it may be removed only after a valid canonical, + rollback or unique valid daily source is established. Invalid canonical, + rollback, duplicate candidate, week mismatch or non-unique rebuild source + fails closed without deleting evidence. A reconciled daily point forces its + ISO-week checkpoint to be rebuilt before finalize continues. When a valid + canonical daily point and its old rollback coexist, defer deleting that daily + rollback until every required weekly reconciliation has succeeded; a weekly + validation or rebuild failure must leave the daily rollback in place. - Write production `last-success.json` and prune source copies only after ARM finalize succeeds. Keep seven automatic source/ARM daily points and four ARM weekly checkpoints; unrelated manual batches are outside automatic retention. @@ -995,6 +1017,14 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ diagnostic. A corrupt same-day candidate leaves the previous archive intact. - Valid same-day retry -> replace only that UTC day's point and rebuild its ISO weekly checkpoint; concurrent finalize -> `flock` rejection. +- Interrupted daily replacement with valid canonical -> remove the verified old + rollback only after rebuilding the weekly point; canonical missing -> restore + the verified rollback; canonical invalid, weekly rebuild failure or candidates + ambiguous -> preserve the remaining rollback evidence and fail. +- Interrupted weekly replacement with valid canonical -> remove safe staging + and verified rollback; canonical missing + valid rollback -> restore it; + staging-only + one valid daily source -> rebuild; any invalid or ambiguous + authority -> preserve all objects and fail. - Missing, corrupt, stale, future or project-mismatched success record -> hard watchdog failure. Fresh build mismatch -> one finding without a failing exit. - Missing/non-regular/symlink/non-`0600` Webhook file -> fail closed without @@ -1025,9 +1055,15 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ runner. A simplified tab-delimited mock is not valid evidence for real mount parsing. - Contract assertions for same-day replacement, corrupt-candidate preservation, - weekly hard-link rebuild, source diagnostic mode, build mismatch finding, - Webhook mode/symlink rejection and group alert behavior without a server - container. Test transport must be explicitly gated and local-only. + weekly hard-link rebuild, daily/weekly interrupted-transaction recovery, + rollback preservation when weekly reconciliation fails, ambiguous-state + preservation, source diagnostic mode, build mismatch finding, Webhook + mode/symlink rejection and group alert behavior without a server container. + Test transport must be explicitly gated and local-only. +- Mock Docker assertions must prove synthetic connection values are absent from + wrapper/backup/restore argv while an inherited `PGPASSWORD` reaches the + mocked container command. Test-fixture `env KEY=value` launch arrays do not + need mechanical rewriting because they contain only synthetic values. - Environment contract assertions for both helpers, both installers, production timers, ARM dead-man timer, explicit project/data/build/SSH/Webhook-file binding, disabled-by-default installation and absence of a literal complete @@ -1044,6 +1080,7 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ ```bash compose exec server send-alert --recipient= rsync -a / backup-host:/ +docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" db pg_dump ... ``` #### Correct @@ -1057,4 +1094,6 @@ rsync -a / backup-host:/ python3 scripts/check-offhost-backup-deadman.py # The runner writes success and prunes only after restricted target finalize. bash scripts/run-automatic-backup.sh run +# The Docker CLI inherits the value without placing it in argv. +PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ... ``` diff --git a/.trellis/tasks/07-27-offhost-backup-final-fixes/check.jsonl b/.trellis/tasks/07-27-offhost-backup-final-fixes/check.jsonl new file mode 100644 index 00000000..9dd3234a --- /dev/null +++ b/.trellis/tasks/07-27-offhost-backup-final-fixes/check.jsonl @@ -0,0 +1 @@ +{"_example": "Fill with {\"file\": \"\", \"reason\": \"\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."} diff --git a/.trellis/tasks/07-27-offhost-backup-final-fixes/design.md b/.trellis/tasks/07-27-offhost-backup-final-fixes/design.md new file mode 100644 index 00000000..e12ad3f2 --- /dev/null +++ b/.trellis/tasks/07-27-offhost-backup-final-fixes/design.md @@ -0,0 +1,75 @@ +# 异机备份最终收尾修复设计 + +## 边界与顺序 + +本任务只修三类已确认缺陷:凭证进入 argv、紧急恢复卡口径漂移、finalize 被强杀后的孤儿事务。receiver 的恢复入口固定为:校验配置 → 取得 `.finalize.lock` → 恢复或拒绝历史孤儿事务 → 校验本次批次 → 发布/取代 → 周检查点 → 保留轮转。没有锁时不得扫描或改变孤儿对象。 + +## 凭证传递合同 + +### 生产 wrapper + +`capture_command` 仍负责将 stdout/stderr 写入 root-only 诊断,但不再把 `env` 当作被执行命令。调用方进入隔离子 shell,在子 shell 内导出 `DATABASE_URL`、Docker DB 身份、备份目录和构建标识,再调用: + +```bash +( + export DATABASE_URL DB_TOOL_MODE DB_CONTAINER POSTGRES_DB POSTGRES_USER \ + POSTGRES_PASSWORD STORAGE_DIR_HOST BACKUP_DIR BACKUP_RUN_ID BUILD_SHA \ + SOURCE_IMAGE_DIGEST + capture_command paired_backup "$ROOT_DIR/scripts/backup-all.sh" +) +``` + +敏感值仍只存在于执行进程环境和内存,不写入命令 argv、诊断、summary 或日志。本任务不宣称对 root 或同执行身份读取 `/proc//environ` 提供额外隔离。 + +### Docker DB 工具 + +生产 Docker 已证明无值 `-e VAR` 能继承客户端环境。三处 Docker 调用统一为: + +```bash +PGPASSWORD="$POSTGRES_PASSWORD" docker exec [-i] -e PGPASSWORD \ + "$DB_CONTAINER" +``` + +`PGPASSWORD` 的值不得出现在 Docker CLI 参数,mock 测试同时检查参数不含合成密码、容器命令能看到继承值。测试夹具用于启动命令的普通 `env KEY=value` 不在本轮机械改写范围。 + +## 日归档孤儿恢复 + +receiver 在锁内扫描 `.daily-replacement-YYYYMMDD.*`。每个孤儿必须是受控 archive 根的一层普通目录且非链接;从其 manifest 取得原 `runId`,执行与正式 finalize 相同的精确文件集、大小和 SHA-256 自校验,并验证日期与孤儿名一致。 + +| 现场 | 含义 | 正确动作 | +| --- | --- | --- | +| 孤儿存在且当日正式归档存在、两者均有效 | 新批次已换入,只缺最后清理 | 保留正式归档,删除孤儿 | +| 孤儿存在且当日正式归档缺失、孤儿有效 | 旧归档已移开,新批次尚未换入 | 按孤儿 manifest 的 `runId` 原子恢复旧正式归档 | +| 两者都在但当日正式归档校验不过 | 无法判断换入结果 | fail closed,两个都保留 | +| 孤儿无效、同日多个正式归档、同日多个孤儿、名称/manifest 日期不符或任一对象为链接 | 状态歧义或受污染 | fail closed,所有对象保留 | + +恢复旧归档或确认新归档后,才允许继续处理本次 partial。任何删除前再次验证目标仍是刚才检查过的严格路径和类型。 + +## 周检查点孤儿恢复 + +周检查点是日归档的硬链接派生物,但仍不得猜测或直接删除。每个 `week-YYYY-Www.new.*` / `.rollback.*` 先按正式周名分组,要求每类最多一个普通目录且非链接。正式周点和 rollback 必须自校验 manifest,且 manifest 日期属于该 ISO 周;`.new` 是从未提交的 staging,内容可能因强杀而不完整,不能作为恢复来源,也不要求内容有效,但只有在另一个有效权威来源已经确定后才可清理。 + +| 现场 | 含义 | 正确动作 | +| --- | --- | --- | +| rollback 存在、正式周点缺失、rollback 有效 | 正式周点已移开,新点尚未换入 | 原子恢复 rollback 为正式周点;`.new` 路径/类型安全时删除 staging,不使用其内容 | +| rollback 存在、正式周点存在且正式点有效 | 新周点已换入,只缺清理 | 保留正式周点,删除 rollback;`.new` 路径/类型安全时删除 staging | +| 仅 `.new` 存在、正式周点存在且有效 | staging 未提交或残留 | 保留正式周点;`.new` 路径/类型安全时删除 staging | +| 仅 `.new` 存在、正式周点缺失 | 没有已提交周点 | 先找到该 ISO 周唯一有效的正式日归档来源;能唯一确定时删除 staging 并从该日归档重建,不能唯一确定时 fail closed 并保留 `.new` | +| 正式周点无效、rollback 无效、多个候选、日期/周不符或任一对象为链接 | 状态歧义或受污染 | fail closed,全部保留 | + +若日归档恢复动作影响某 ISO 周,必须从最终选定且已验证的正式日归档重建该周检查点后才算 reconciliation 成功。周点恢复/重建失败时不删除日归档 rollback。 + +## 失败传播与可观测性 + +reconciliation 任一歧义返回非零,receiver 不处理本次 partial。源端 SSH finalize 随即在 `offhost_finalize` 阶段失败,`capture_command` 保留 `0600` 诊断,`on_error` 通过独立企微群机器人发送脱敏类别,且不写 `last-success.json`、不清理源端批次。若故障持续,ARM dead-man 在超龄窗口独立告警。该双层链路已经真实验证,本任务只用合成测试证明返回值和源端状态传播,不再次制造真实断链。 + +## 文档与单管理员恢复卡 + +runbook 标题改为“紧急恢复卡(单管理员离线版)”,正文只保留唯一系统管理员、ARM 隔离影子恢复、真实生产恢复另行批准和不粘贴敏感内容的约束。复制到 ARM 与离线副本是后续可选运维动作,不写死真实路径、不进入代码验收。 + +## 兼容、部署与回滚 + +- 不改变批次、manifest、receiver forced-command、timer、保留数量或告警 payload 合同。 +- 本地提交后先停在外部门禁;推送、生产 runner/DB 脚本部署、ARM receiver 更新分别请求批准。 +- 代码回滚只回滚本任务提交;已存在的正式归档、周检查点、partial 和孤儿对象均不删除。 +- 完成后归档本任务并结束运维加固主线。 diff --git a/.trellis/tasks/07-27-offhost-backup-final-fixes/implement.jsonl b/.trellis/tasks/07-27-offhost-backup-final-fixes/implement.jsonl new file mode 100644 index 00000000..9dd3234a --- /dev/null +++ b/.trellis/tasks/07-27-offhost-backup-final-fixes/implement.jsonl @@ -0,0 +1 @@ +{"_example": "Fill with {\"file\": \"\", \"reason\": \"\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."} diff --git a/.trellis/tasks/07-27-offhost-backup-final-fixes/implement.md b/.trellis/tasks/07-27-offhost-backup-final-fixes/implement.md new file mode 100644 index 00000000..ae773f1c --- /dev/null +++ b/.trellis/tasks/07-27-offhost-backup-final-fixes/implement.md @@ -0,0 +1,55 @@ +# 异机备份最终收尾修复实施计划 + +## 实施步骤 + +- [x] 先扩展合成测试:记录 mock Docker argv 与继承环境,证明四条凭证路径不含合成密码/完整连接串且容器能收到 `PGPASSWORD`。 +- [x] 修改 `run-automatic-backup.sh`,在隔离子 shell内导出成对备份变量后调用 `capture_command`,移除生产路径 `env KEY=value` argv。 +- [x] 修改 `backup-db.sh` 和 `restore-db.sh` 三处 Docker 调用,使用前缀环境赋值配合无值 `-e PGPASSWORD`;不批量改写测试夹具。 +- [x] 为 receiver 增加锁内 self-validation 与 orphan reconciliation helper,严格实现 `design.md` 的日归档和周检查点状态表。 +- [x] 增加孤儿测试矩阵:明确恢复、明确清理、周点重建、非法类型/链接/重复候选/校验失败全量保留,以及 fail-closed 到源端状态的合成传播。 +- [x] 更新 runbook 恢复卡标题和正文为单管理员口径;同步 `.trellis/spec/hyb-oa/deployment-trialrun.md` 的凭证 argv 与中断恢复稳定合同,不新建平行文档。 +- [x] 运行聚焦与全量相关门禁,完成只读差异复核;不连接真实告警、不写 ARM/生产、不执行 migration。 +- [x] 经用户确认后提交中文 commit;推送和生产/ARM 部署另行批准。 +- [ ] 部署验证完成后归档本任务,硬收口运维主线并返回产品 backlog;不得因本轮审查再新增运维加固 task。 + +## 预计文件边界 + +- `scripts/run-automatic-backup.sh` +- `scripts/backup-db.sh` +- `scripts/restore-db.sh` +- `scripts/receive-offhost-backup.sh` +- `scripts/qa/run-automatic-backup-contract.sh` +- 必要时扩展现有 `scripts/qa/run-backup-restore-drill.sh` 的 argv 断言,不新增平行 runner。 +- `deploy/trialrun/runbook.md` +- `.trellis/spec/hyb-oa/deployment-trialrun.md` +- 当前 Trellis task 三份规划/证据文件。 + +## 验证命令 + +- `bash -n scripts/run-automatic-backup.sh scripts/backup-db.sh scripts/restore-db.sh scripts/receive-offhost-backup.sh scripts/qa/run-automatic-backup-contract.sh scripts/qa/run-backup-restore-drill.sh` +- `bash scripts/qa/run-automatic-backup-contract.sh` +- `bash scripts/qa/run-backup-restore-drill.sh` +- `node deploy/trialrun/validate-env-contract.js` +- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-27-offhost-backup-final-fixes` +- `node scripts/qa/check-trellis-governance.mjs` +- `node scripts/qa/check-sensitive-content.mjs` +- `node scripts/qa/audit-docs.mjs --check-registry` +- `node scripts/qa/check-process-contract.mjs --baseline --json` +- `git diff --check` + +## 外部门禁与回滚点 + +- 生产 Docker 无值 `-e VAR` 继承只读实测已完成;实施阶段不重复读取生产配置或凭证。 +- 推送当前 3 个未推送提交与本任务提交,需要提交完成后另行批准。 +- 更新生产自动备份脚本、ARM receiver、手工触发真实备份或复制恢复卡到 ARM,均需分别明确批准。 +- 生产部署前只读确认活动构建、62 项 migration、四组 timer 和最近成功状态;本任务没有 migration。 +- 任一 orphan 合成负例发生对象减少、正式恢复点丢失或歧义状态被自动选择,立即停止,不进入部署。 +- 回滚只回退本任务脚本/文档,不清理现场备份、partial、rollback 或已归档 task。 + +## 本地实施与验证证据(2026-07-27) + +- 凭证传递已覆盖 wrapper、nightly Docker 备份、Docker 影子确认和恢复;mock 捕获的 argv 不含合成连接串/口令,容器命令仍收到继承的 `PGPASSWORD`。 +- receiver 合同共 73 项通过。除原有同日取代、保留轮转、dead-man 和失败传播外,新增日/周孤儿恢复、重复/链接/损坏歧义 fail closed,以及“周 reconciliation 失败时日 rollback 不提前删除”的回归断言。 +- `run-backup-restore-drill.sh` 通过,隔离合成 RTO 为 9 秒;`validate-env-contract.js` 返回 `[R26][PASS]`。 +- shell 语法、task validate、Trellis governance、敏感内容扫描、文档注册审计、流程 baseline 和 `git diff --check` 均通过。文档审计仅报告仓库既有 22 个未注册 docs 文件警告,本任务未新增 docs 文件。 +- 只读复核发现并修复了“先删日 rollback、后重建周点”的顺序缺陷;稳定规则已同步到 `deployment-trialrun.md`。本地过程未连接真实企微、ARM 或生产,也未执行 migration。 diff --git a/.trellis/tasks/07-27-offhost-backup-final-fixes/prd.md b/.trellis/tasks/07-27-offhost-backup-final-fixes/prd.md new file mode 100644 index 00000000..c0c53c9d --- /dev/null +++ b/.trellis/tasks/07-27-offhost-backup-final-fixes/prd.md @@ -0,0 +1,62 @@ +# 异机备份最终收尾修复 + +## Goal + +在不扩建运维平台的前提下,完成自动异机备份最后一轮有界修复:让生产热路径和 Docker 备份/恢复入口不再把数据库凭证放入进程 argv;让紧急恢复卡与唯一系统管理员的事实一致;让 ARM finalize 在被强杀后能够安全恢复明确的孤儿事务,并在歧义状态下保留全部现场、失败告警而非静默停摆。 + +## Background + +- `origin/main` 与当前生产活动构建均为 `a0006f0`;`2526052` 已推送,生产已有 62 项成功 migration,手机号 nullable migration 不需要重复部署。 +- 当前待推送提交只有 `de35ab8`、`95b51e6`、`f1e8cf1`;本任务完成前不单独推送,避免本地与远端继续形成多个收尾批次。 +- `scripts/run-automatic-backup.sh:344` 使用 `env KEY=value` 调用成对备份,完整 `DATABASE_URL` 和数据库口令会进入 `env` 进程 argv。 +- `scripts/backup-db.sh:74`、`scripts/restore-db.sh:101`、`scripts/restore-db.sh:113` 使用 `docker exec -e PGPASSWORD=value`,其中 nightly 自动备份会经过 `backup-db.sh`,因此只修 wrapper 不能关闭热路径暴露。 +- 生产 Docker `20.10.24` 已用无敏感合成变量只读实测:`VAR=value docker exec -e VAR ...` 能把客户端环境值继承给容器命令;测试只记录 `inherited=yes`,未读取或输出数据库凭证。 +- `deploy/trialrun/runbook.md` 的紧急恢复卡仍写“复制到公司知识库”和“第二名内部人员”,与已经批准的唯一系统管理员口径矛盾。 +- receiver 已有 `flock` 和正常错误回滚,但 `SIGKILL` 无法执行 shell 清理;强杀窗口可能留下 `.daily-replacement-*`、`.new.*` 或 `.rollback.*`,下一次 finalize 当前没有启动恢复逻辑。 + +## Scope + +- 修复 wrapper 与三处 Docker DB 工具调用的凭证 argv 暴露,并增加合成回归断言。 +- 增加 receiver 锁内日归档/周检查点孤儿事务恢复及 fail-closed 测试矩阵。 +- 同步单管理员紧急恢复卡和部署规约;完成本地验证、提交与后续受控部署后归档运维主线。 + +## Requirements + +- 四处已确认的凭证调用必须改为环境继承,敏感值不得出现在调用命令 argv:wrapper 在隔离子 shell 中导出所需变量后调用 `capture_command`;Docker 入口使用 `PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD ...`,不得使用 `env DATABASE_URL=...` 或 `-e PGPASSWORD=...`。 +- 契约测试中的 `env KEY=value` 是合成夹具,不含真实凭证,不做无意义的批量改写;新增断言必须使用合成标记证明敏感值不出现在 mock 捕获的 argv。 +- 紧急恢复卡标题、负责人和季度动作必须统一为单管理员口径:恢复卡保存在现有 runbook,由唯一系统管理员维护并按季度在 ARM 隔离目标执行影子恢复;不虚构知识库副本或第二人门禁。 +- finalize 必须在取得现有独占锁之后、处理本次 partial 之前,扫描严格命名的日归档和周检查点孤儿事务;正式点和 rollback 只有在普通目录、非符号链接、位于受控根目录且通过 manifest/文件集/大小/SHA-256 自校验后才可作为权威来源。`.new` 永远不是权威来源,只能在严格路径/类型校验且另一个有效权威来源已经确定后清理。 +- `.daily-replacement-*` 的判定必须遵守 `design.md` 的日归档状态表:孤儿与有效当日归档同时存在时删除孤儿;孤儿存在但当日归档缺失时从孤儿 manifest 还原原归档名;当日归档校验失败、存在多个候选或任何身份歧义时 fail closed,两个对象均保留。 +- 周检查点 `.new.*`/`.rollback.*` 必须遵守 `design.md` 的对称状态表:rollback 在正式周检查点缺失时优先恢复;正式周检查点有效时临时对象才可清理;只有 staging 且正式点缺失时,必须先找到同 ISO 周唯一、有效的日归档来源再重建;歧义或校验失败一律保留现场并失败。 +- 自动清理必须逐目标执行严格路径、类型和命名校验;不得用宽泛 glob 删除,不得触碰正式日归档、正式周检查点、手工批次或最后一个有效恢复点。 +- orphan 歧义导致 receiver 非零时,源端必须继续在 `offhost_finalize` 阶段失败、保留源批次和 `0600` 诊断并走独立群机器人告警;ARM dead-man 继续在超龄窗口独立告警。本任务复用已真实验证的链路,不再人为制造第二次真实断链。 +- 恢复卡复制到 ARM 和离线副本属于可选纯运维缓解项,不是代码验收阻断;任何 ARM 写入、推送、生产部署或真实备份执行仍需另行批准。 +- 本任务完成后硬收口运维加固,不继续新增备份平台、告警能力、保留层级或恢复自动化,后续返回产品 backlog。 + +## Acceptance Criteria + +- [x] 合成 argv 测试证明 wrapper、nightly Docker 备份、Docker 影子确认和 Docker 影子恢复四条路径的命令参数均不包含合成密码或完整连接串,同时容器命令仍收到 `PGPASSWORD`。 +- [x] `run-automatic-backup.sh` 不再出现生产路径 `capture_command ... env DATABASE_URL=...`;`backup-db.sh`/`restore-db.sh` 不再出现 `docker exec ... -e PGPASSWORD=value`。 +- [x] 恢复卡正文、标题、负责人和季度动作与《管理员手册》单管理员口径一致,且仍保留真实生产恢复的单独批准门禁。 +- [x] 日归档测试覆盖“孤儿+有效正式点则清理孤儿”“孤儿+缺正式点则恢复旧点”“孤儿+无效正式点则两个都保留并失败”“重复/非法/链接候选则失败且不删除”。 +- [x] 周检查点测试覆盖 `.new`、`.rollback`、正式点存在/缺失、可唯一重建和歧义失败;任何成功恢复后只保留一个正式周检查点,任何歧义失败后现场对象不减少。 +- [x] orphan fail closed 通过本地合成链路证明 finalize 非零、源端不写成功状态、不执行源端保留清理且诊断为 `0600`;不重复执行真实企微断链验收。 +- [x] 现有同日取代、失败回滚、7 日 + 4 周保留、dead-man、备份/恢复往返、环境合同和敏感扫描均无回退。 +- [x] task、Trellis governance、文档注册/流程合同和 `git diff --check` 通过;只有经确认的任务文件进入提交。 + +## Out of Scope + +- 不新增数据库 migration,不重复部署 `2526052`,不修改生产业务数据。 +- 不重测已经通过的真实企微失败、重复降噪和恢复通知链路。 +- 不自动执行恢复,不删除 ARM 正式归档或手工批次,不改变 7 日 + 4 周保留策略。 +- 不改写仅含合成值的测试夹具 `env` 调用,不做全仓无关安全重构。 +- 不在本任务直接推送、部署、更新 ARM 脚本或复制恢复卡;这些动作分别保持外部门禁。 +- 不继续新增运维能力;本任务之后不因本轮审查再派生新的运维加固 task。 + +## Test Plan + +- Shell 语法检查和 `bash scripts/qa/run-automatic-backup-contract.sh`,扩展 mock argv/环境继承与 orphan 恢复矩阵。 +- `bash scripts/qa/run-backup-restore-drill.sh`,覆盖 Docker 备份与影子恢复入口无回退。 +- `node deploy/trialrun/validate-env-contract.js`。 +- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-27-offhost-backup-final-fixes`。 +- Trellis governance、敏感扫描、文档注册、流程合同和 `git diff --check`。 diff --git a/.trellis/tasks/07-27-offhost-backup-final-fixes/task.json b/.trellis/tasks/07-27-offhost-backup-final-fixes/task.json new file mode 100644 index 00000000..c7aaae02 --- /dev/null +++ b/.trellis/tasks/07-27-offhost-backup-final-fixes/task.json @@ -0,0 +1,26 @@ +{ + "id": "offhost-backup-final-fixes", + "name": "offhost-backup-final-fixes", + "title": "异机备份最终收尾修复", + "description": "修复自动异机备份凭证进入进程 argv、单管理员恢复卡漂移和 finalize 强杀后孤儿事务恢复,完成后硬收口运维主线。", + "status": "in_progress", + "dev_type": null, + "scope": "scripts/run-automatic-backup.sh; scripts/backup-db.sh; scripts/restore-db.sh; scripts/receive-offhost-backup.sh; automatic-backup contracts; trialrun runbook/spec", + "package": null, + "priority": "P0", + "creator": "shunli", + "assignee": "shunli", + "createdAt": "2026-07-27", + "completedAt": null, + "branch": null, + "base_branch": "main", + "worktree_path": null, + "commit": null, + "pr_url": null, + "subtasks": [], + "children": [], + "parent": null, + "relatedFiles": [], + "notes": "", + "meta": {} +} \ No newline at end of file diff --git a/deploy/trialrun/runbook.md b/deploy/trialrun/runbook.md index 09d6502b..2a7e3d1b 100644 --- a/deploy/trialrun/runbook.md +++ b/deploy/trialrun/runbook.md @@ -2,9 +2,9 @@ > 目标:按本文档在一台新机器完成 **启动 → 迁移 → 初始化管理员 → 验收 → 备份 → 恢复 → 回滚** 演练。 -## 紧急恢复卡(复制到公司知识库的一页版) +## 紧急恢复卡(单管理员离线版) -适用情形:生产主机不可恢复、数据库或附件损坏,且已由负责人批准进入事故恢复。主操作人为系统管理员,复核人为另一名已完成季度演练的内部人员;两人均不得在聊天或工单中粘贴凭证、业务正文、真实目录或原始日志。 +适用情形:生产主机不可恢复、数据库或附件损坏,且已由负责人批准进入事故恢复。公司当前只有一名系统管理员,由该管理员维护并执行本卡;不得在聊天或工单中粘贴凭证、业务正文、真实目录或原始日志。 1. 先暂停外部入口和新业务写入,不删除故障主机、旧数据目录或 ARM 归档;记录故障时间、最近成功批次编号类别和恢复负责人。 2. 在 ARM 的只读归档中选择最近一个 `complete` 批次,复核三文件、manifest、SHA-256 与来源标识;先复制到隔离恢复工作区,不直接对归档原件操作。 @@ -21,7 +21,7 @@ curl -fsS 4. 复核 `/ready`、运行构建标识、成功 migration 数、组织/权限/报销/付款等关键表计数、附件普通文件数及 DB/附件关联;任何一项不一致都停止切换。 5. 真实生产恢复仍是 **EXTERNAL_GATED**:另建或更新获批 Trellis task,冻结新数据目录、回滚点和停机窗口后才切换。恢复后保留 ARM 归档和故障主机原数据,记录 RTO/RPO 与结论。 -本卡必须由第二名内部人员每季度只读走查一次,并实际在 ARM 隔离目标完成人工影子恢复;自动 timer 只提醒,不自动恢复。 +唯一系统管理员必须每季度按本卡在 ARM 隔离目标完成人工影子恢复;自动 timer 只提醒,不自动恢复。可在 ARM 和离线介质各保留一份脱敏副本,但不得写入凭证、真实目录或原始日志。 ## 0A. 如果官网 `hyb-web` 已在同机运行(推荐方案) - 官网继续保持 `hyinda.com / www.hyinda.com` 静态托管,不要改官网目录和配置。 diff --git a/scripts/backup-db.sh b/scripts/backup-db.sh index 2161f8cd..1f8044c6 100755 --- a/scripts/backup-db.sh +++ b/scripts/backup-db.sh @@ -71,7 +71,7 @@ if [[ -z "$POSTGRES_DB" || -z "$POSTGRES_USER" || -z "$POSTGRES_PASSWORD" ]]; th exit 1 fi -if ! docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \ +if ! PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD "$DB_CONTAINER" \ pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" --format=c --no-owner --no-acl \ >"$OUT_FILE"; then rm -f -- "$OUT_FILE" diff --git a/scripts/qa/run-automatic-backup-contract.sh b/scripts/qa/run-automatic-backup-contract.sh index 4736ccec..71c49a3b 100755 --- a/scripts/qa/run-automatic-backup-contract.sh +++ b/scripts/qa/run-automatic-backup-contract.sh @@ -128,6 +128,162 @@ expect_failure "receiver rolls back the daily archive when weekly checkpoint pub [[ -d "$ROLLBACK_LAYOUT/archive/$CURRENT_BATCH" && ! -e "$ROLLBACK_LAYOUT/archive/$SECOND_BATCH" ]] pass "receiver restores the prior daily point after a later replacement step fails" +CURRENT_WEEK="$(python3 - "$CURRENT_RUN_ID" <<'PYTHON' +from datetime import datetime +import sys +day = datetime.strptime(sys.argv[1][5:13], "%Y%m%d").date().isocalendar() +print(f"week-{day.year:04d}-W{day.week:02d}") +PYTHON +)" + +make_corrupt_partial() { + local layout="$1" + local run_id="$2" + local batch_name="hyb-oa-backup-$run_id" + make_batch "$layout/inbox/${batch_name}.partial" "$run_id" + printf 'corrupt candidate\n' >>"$layout/inbox/${batch_name}.partial/database.dump" + sha256sum "$layout/inbox/${batch_name}.partial/manifest.json" | awk '{print $1}' +} + +DAILY_COMPLETE_LAYOUT="$TEST_DIR/receiver-daily-orphan-complete" +make_layout "$DAILY_COMPLETE_LAYOUT" +make_batch "$DAILY_COMPLETE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.101" "$CURRENT_RUN_ID" +make_batch "$DAILY_COMPLETE_LAYOUT/archive/$SECOND_BATCH" "$SECOND_RUN_ID" +DAILY_COMPLETE_SHA="$(make_corrupt_partial "$DAILY_COMPLETE_LAYOUT" "$THIRD_RUN_ID")" +expect_failure "receiver cleans a valid daily rollback after the replacement was published" \ + receiver "$DAILY_COMPLETE_LAYOUT" finalize-test "$THIRD_RUN_ID" "$DAILY_COMPLETE_SHA" +[[ ! -e "$DAILY_COMPLETE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.101" \ + && -d "$DAILY_COMPLETE_LAYOUT/archive/$SECOND_BATCH" ]] +[[ -d "$DAILY_COMPLETE_LAYOUT/weekly/$CURRENT_WEEK" ]] +pass "receiver keeps the valid published daily point and rebuilds its weekly checkpoint" + +DAILY_WEEKLY_FAIL_LAYOUT="$TEST_DIR/receiver-daily-orphan-weekly-failure" +make_layout "$DAILY_WEEKLY_FAIL_LAYOUT" +DAILY_WEEKLY_FAIL_ORPHAN="$DAILY_WEEKLY_FAIL_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.107" +make_batch "$DAILY_WEEKLY_FAIL_ORPHAN" "$CURRENT_RUN_ID" +make_batch "$DAILY_WEEKLY_FAIL_LAYOUT/archive/$SECOND_BATCH" "$SECOND_RUN_ID" +printf 'invalid weekly target\n' >"$DAILY_WEEKLY_FAIL_LAYOUT/weekly/$CURRENT_WEEK" +DAILY_WEEKLY_FAIL_SHA="$(make_corrupt_partial "$DAILY_WEEKLY_FAIL_LAYOUT" "$THIRD_RUN_ID")" +expect_failure "receiver keeps a daily rollback until weekly reconciliation succeeds" \ + receiver "$DAILY_WEEKLY_FAIL_LAYOUT" finalize-test "$THIRD_RUN_ID" "$DAILY_WEEKLY_FAIL_SHA" +[[ -d "$DAILY_WEEKLY_FAIL_ORPHAN" \ + && -d "$DAILY_WEEKLY_FAIL_LAYOUT/archive/$SECOND_BATCH" \ + && -f "$DAILY_WEEKLY_FAIL_LAYOUT/weekly/$CURRENT_WEEK" ]] +pass "receiver preserves the daily rollback when weekly reconciliation fails" + +DAILY_RESTORE_LAYOUT="$TEST_DIR/receiver-daily-orphan-restore" +make_layout "$DAILY_RESTORE_LAYOUT" +make_batch "$DAILY_RESTORE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.102" "$CURRENT_RUN_ID" +DAILY_RESTORE_SHA="$(make_corrupt_partial "$DAILY_RESTORE_LAYOUT" "$SECOND_RUN_ID")" +expect_failure "receiver restores a valid daily rollback when the published archive is absent" \ + receiver "$DAILY_RESTORE_LAYOUT" finalize-test "$SECOND_RUN_ID" "$DAILY_RESTORE_SHA" +[[ -d "$DAILY_RESTORE_LAYOUT/archive/$CURRENT_BATCH" \ + && ! -e "$DAILY_RESTORE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.102" ]] +[[ -d "$DAILY_RESTORE_LAYOUT/weekly/$CURRENT_WEEK" ]] +pass "receiver restores the last daily point before rejecting the next corrupt candidate" + +DAILY_AMBIGUOUS_LAYOUT="$TEST_DIR/receiver-daily-orphan-ambiguous" +make_layout "$DAILY_AMBIGUOUS_LAYOUT" +make_batch "$DAILY_AMBIGUOUS_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.103" "$CURRENT_RUN_ID" +make_batch "$DAILY_AMBIGUOUS_LAYOUT/archive/$SECOND_BATCH" "$SECOND_RUN_ID" +printf 'corruption\n' >>"$DAILY_AMBIGUOUS_LAYOUT/archive/$SECOND_BATCH/database.dump" +make_batch "$DAILY_AMBIGUOUS_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID" +DAILY_AMBIGUOUS_SHA="$(sha256sum "$DAILY_AMBIGUOUS_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')" +expect_failure "receiver fails closed when a daily rollback and invalid published archive coexist" \ + receiver "$DAILY_AMBIGUOUS_LAYOUT" finalize-test "$THIRD_RUN_ID" "$DAILY_AMBIGUOUS_SHA" +[[ -d "$DAILY_AMBIGUOUS_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.103" \ + && -d "$DAILY_AMBIGUOUS_LAYOUT/archive/$SECOND_BATCH" ]] +pass "receiver preserves both daily objects when replacement state is ambiguous" + +DAILY_DUPLICATE_LAYOUT="$TEST_DIR/receiver-daily-orphan-duplicate" +make_layout "$DAILY_DUPLICATE_LAYOUT" +make_batch "$DAILY_DUPLICATE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.104" "$CURRENT_RUN_ID" +make_batch "$DAILY_DUPLICATE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.105" "$CURRENT_RUN_ID" +make_batch "$DAILY_DUPLICATE_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID" +DAILY_DUPLICATE_SHA="$(sha256sum "$DAILY_DUPLICATE_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')" +expect_failure "receiver fails closed on duplicate daily rollback orphans" \ + receiver "$DAILY_DUPLICATE_LAYOUT" finalize-test "$SECOND_RUN_ID" "$DAILY_DUPLICATE_SHA" +[[ -d "$DAILY_DUPLICATE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.104" \ + && -d "$DAILY_DUPLICATE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.105" ]] +pass "receiver preserves duplicate daily rollback evidence without choosing one" + +DAILY_LINK_LAYOUT="$TEST_DIR/receiver-daily-orphan-link" +make_layout "$DAILY_LINK_LAYOUT" +make_batch "$DAILY_LINK_LAYOUT/archive/$CURRENT_BATCH" "$CURRENT_RUN_ID" +ln -s "$DAILY_LINK_LAYOUT/archive/$CURRENT_BATCH" \ + "$DAILY_LINK_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.106" +make_batch "$DAILY_LINK_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID" +DAILY_LINK_SHA="$(sha256sum "$DAILY_LINK_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')" +expect_failure "receiver rejects a symbolic-link daily rollback orphan" \ + receiver "$DAILY_LINK_LAYOUT" finalize-test "$SECOND_RUN_ID" "$DAILY_LINK_SHA" +[[ -L "$DAILY_LINK_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.106" \ + && -d "$DAILY_LINK_LAYOUT/archive/$CURRENT_BATCH" ]] +pass "receiver leaves linked daily rollback evidence untouched" + +WEEKLY_ROLLBACK_LAYOUT="$TEST_DIR/receiver-weekly-orphan-rollback" +make_layout "$WEEKLY_ROLLBACK_LAYOUT" +make_batch "$WEEKLY_ROLLBACK_LAYOUT/weekly/${CURRENT_WEEK}.rollback.201" "$CURRENT_RUN_ID" +WEEKLY_ROLLBACK_SHA="$(make_corrupt_partial "$WEEKLY_ROLLBACK_LAYOUT" "$SECOND_RUN_ID")" +expect_failure "receiver restores a weekly rollback when the canonical checkpoint is absent" \ + receiver "$WEEKLY_ROLLBACK_LAYOUT" finalize-test "$SECOND_RUN_ID" "$WEEKLY_ROLLBACK_SHA" +[[ -d "$WEEKLY_ROLLBACK_LAYOUT/weekly/$CURRENT_WEEK" \ + && ! -e "$WEEKLY_ROLLBACK_LAYOUT/weekly/${CURRENT_WEEK}.rollback.201" ]] +pass "receiver restores the authoritative weekly rollback before the next finalize" + +WEEKLY_COMPLETE_LAYOUT="$TEST_DIR/receiver-weekly-orphan-complete" +make_layout "$WEEKLY_COMPLETE_LAYOUT" +make_batch "$WEEKLY_COMPLETE_LAYOUT/weekly/$CURRENT_WEEK" "$SECOND_RUN_ID" +make_batch "$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.rollback.202" "$CURRENT_RUN_ID" +mkdir "$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.new.202" +printf 'interrupted staging\n' >"$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.new.202/partial" +WEEKLY_COMPLETE_SHA="$(make_corrupt_partial "$WEEKLY_COMPLETE_LAYOUT" "$THIRD_RUN_ID")" +expect_failure "receiver cleans weekly rollback and staging after a valid checkpoint was published" \ + receiver "$WEEKLY_COMPLETE_LAYOUT" finalize-test "$THIRD_RUN_ID" "$WEEKLY_COMPLETE_SHA" +[[ -d "$WEEKLY_COMPLETE_LAYOUT/weekly/$CURRENT_WEEK" \ + && ! -e "$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.rollback.202" \ + && ! -e "$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.new.202" ]] +pass "receiver treats weekly staging as non-authoritative after a valid publish" + +WEEKLY_REBUILD_LAYOUT="$TEST_DIR/receiver-weekly-orphan-rebuild" +make_layout "$WEEKLY_REBUILD_LAYOUT" +make_batch "$WEEKLY_REBUILD_LAYOUT/archive/$CURRENT_BATCH" "$CURRENT_RUN_ID" +mkdir "$WEEKLY_REBUILD_LAYOUT/weekly/${CURRENT_WEEK}.new.203" +printf 'interrupted staging\n' >"$WEEKLY_REBUILD_LAYOUT/weekly/${CURRENT_WEEK}.new.203/partial" +WEEKLY_REBUILD_SHA="$(make_corrupt_partial "$WEEKLY_REBUILD_LAYOUT" "$SECOND_RUN_ID")" +expect_failure "receiver rebuilds a missing weekly point from one valid daily archive" \ + receiver "$WEEKLY_REBUILD_LAYOUT" finalize-test "$SECOND_RUN_ID" "$WEEKLY_REBUILD_SHA" +[[ -d "$WEEKLY_REBUILD_LAYOUT/weekly/$CURRENT_WEEK" \ + && ! -e "$WEEKLY_REBUILD_LAYOUT/weekly/${CURRENT_WEEK}.new.203" ]] +[[ "$(stat -c %i "$WEEKLY_REBUILD_LAYOUT/weekly/$CURRENT_WEEK/database.dump")" \ + == "$(stat -c %i "$WEEKLY_REBUILD_LAYOUT/archive/$CURRENT_BATCH/database.dump")" ]] +pass "receiver rebuilds the weekly checkpoint from the unique valid daily source" + +WEEKLY_AMBIGUOUS_LAYOUT="$TEST_DIR/receiver-weekly-orphan-ambiguous" +make_layout "$WEEKLY_AMBIGUOUS_LAYOUT" +make_batch "$WEEKLY_AMBIGUOUS_LAYOUT/archive/$CURRENT_BATCH" "$CURRENT_RUN_ID" +make_batch "$WEEKLY_AMBIGUOUS_LAYOUT/archive/$SECOND_BATCH" "$SECOND_RUN_ID" +mkdir "$WEEKLY_AMBIGUOUS_LAYOUT/weekly/${CURRENT_WEEK}.new.204" +make_batch "$WEEKLY_AMBIGUOUS_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID" +WEEKLY_AMBIGUOUS_SHA="$(sha256sum "$WEEKLY_AMBIGUOUS_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')" +expect_failure "receiver fails closed when weekly staging has multiple daily sources" \ + receiver "$WEEKLY_AMBIGUOUS_LAYOUT" finalize-test "$THIRD_RUN_ID" "$WEEKLY_AMBIGUOUS_SHA" +[[ -d "$WEEKLY_AMBIGUOUS_LAYOUT/weekly/${CURRENT_WEEK}.new.204" \ + && ! -e "$WEEKLY_AMBIGUOUS_LAYOUT/weekly/$CURRENT_WEEK" ]] +pass "receiver preserves ambiguous weekly staging without choosing a daily source" + +WEEKLY_INVALID_LAYOUT="$TEST_DIR/receiver-weekly-orphan-invalid" +make_layout "$WEEKLY_INVALID_LAYOUT" +make_batch "$WEEKLY_INVALID_LAYOUT/weekly/$CURRENT_WEEK" "$SECOND_RUN_ID" +printf 'corruption\n' >>"$WEEKLY_INVALID_LAYOUT/weekly/$CURRENT_WEEK/database.dump" +make_batch "$WEEKLY_INVALID_LAYOUT/weekly/${CURRENT_WEEK}.rollback.205" "$CURRENT_RUN_ID" +make_batch "$WEEKLY_INVALID_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID" +WEEKLY_INVALID_SHA="$(sha256sum "$WEEKLY_INVALID_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')" +expect_failure "receiver fails closed when a weekly rollback and invalid canonical coexist" \ + receiver "$WEEKLY_INVALID_LAYOUT" finalize-test "$THIRD_RUN_ID" "$WEEKLY_INVALID_SHA" +[[ -d "$WEEKLY_INVALID_LAYOUT/weekly/$CURRENT_WEEK" \ + && -d "$WEEKLY_INVALID_LAYOUT/weekly/${CURRENT_WEEK}.rollback.205" ]] +pass "receiver preserves both weekly objects when checkpoint state is ambiguous" + negative_receiver_case() { local label="$1" local mutation="$2" @@ -241,7 +397,10 @@ printf 'backup-alias ssh-ed25519 fixture\n' >"$TEST_DIR/known-hosts" BUILD_SHA="abcdef1234567890abcdef1234567890abcdef12" VITE_BUILD_SHA="${BUILD_SHA:0:12}" MOCK_ALERT_LOG="$TEST_DIR/mock-alert.log" +MOCK_DOCKER_ARGV_LOG="$TEST_DIR/mock-docker-argv.log" +MOCK_DOCKER_ENV_LOG="$TEST_DIR/mock-docker-env.log" export MOCK_PG="$WRAPPER_PG" MOCK_STORAGE="$WRAPPER_STORAGE" MOCK_BUILD_SHA="$BUILD_SHA" MOCK_ALERT_LOG +export MOCK_DOCKER_ARGV_LOG MOCK_DOCKER_ENV_LOG cat >"$MOCK_BIN/docker" <<'MOCK' #!/usr/bin/env bash @@ -278,8 +437,13 @@ if [[ "${1:-}" == "inspect" ]]; then exit 0 fi if [[ "${1:-}" == "exec" ]]; then + printf '%q ' "$@" >>"$MOCK_DOCKER_ARGV_LOG" + printf '\n' >>"$MOCK_DOCKER_ARGV_LOG" + printf 'PGPASSWORD=%s\n' "${PGPASSWORD:-}" >>"$MOCK_DOCKER_ENV_LOG" if [[ " $* " == *" pg_dump "* ]]; then printf 'synthetic-dump\n'; exit 0; fi + if [[ " $* " == *" psql "* ]]; then printf '%s\n' "${POSTGRES_DB:-hyb_oa_restore_shadow}"; exit 0; fi if [[ " $* " == *" pg_restore --list "* ]]; then cat >/dev/null; exit 0; fi + if [[ " $* " == *" pg_restore "* ]]; then cat >/dev/null; exit 0; fi fi exit 1 MOCK @@ -365,6 +529,54 @@ pass "wrapper failure summary is category-only and redacted" [[ -f "$WRAPPER_STATE/last-error.log" && "$(stat -c %a "$WRAPPER_STATE/last-error.log")" == "600" ]] grep -Fq 'synthetic transfer diagnostic' "$WRAPPER_STATE/last-error.log" pass "wrapper retains the latest command diagnostic locally with mode 0600" +[[ -s "$MOCK_DOCKER_ARGV_LOG" && -s "$MOCK_DOCKER_ENV_LOG" ]] +! grep -Fq 'fixture' "$MOCK_DOCKER_ARGV_LOG" +! grep -Fq 'postgresql://' "$MOCK_DOCKER_ARGV_LOG" +grep -Fq 'PGPASSWORD=fixture' "$MOCK_DOCKER_ENV_LOG" +pass "wrapper and nightly Docker backup keep connection secrets out of captured argv" + +: >"$MOCK_DOCKER_ARGV_LOG" +: >"$MOCK_DOCKER_ENV_LOG" +DB_TOOL_MODE=docker DATABASE_URL='postgresql://synthetic:argv-secret@db:5432/hyb_oa_restore_shadow' \ + DB_CONTAINER=db-id POSTGRES_DB=hyb_oa_restore_shadow POSTGRES_USER=synthetic \ + POSTGRES_PASSWORD=argv-secret OUT_FILE="$TEST_DIR/argv-backup.dump" PATH="$MOCK_BIN:$PATH" \ + "$ROOT_DIR/scripts/backup-db.sh" >/dev/null +printf 'synthetic restore dump\n' >"$TEST_DIR/argv-restore.dump" +DB_TOOL_MODE=docker DATABASE_URL='postgresql://synthetic:argv-secret@db:5432/hyb_oa_restore_shadow' \ + DB_CONTAINER=db-id POSTGRES_DB=hyb_oa_restore_shadow POSTGRES_USER=synthetic \ + POSTGRES_PASSWORD=argv-secret RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE=hyb_oa_restore_shadow \ + PATH="$MOCK_BIN:$PATH" "$ROOT_DIR/scripts/restore-db.sh" "$TEST_DIR/argv-restore.dump" >/dev/null +! grep -Fq 'argv-secret' "$MOCK_DOCKER_ARGV_LOG" +grep -Fq 'exec -e PGPASSWORD db-id pg_dump' "$MOCK_DOCKER_ARGV_LOG" +grep -Fq 'exec -e PGPASSWORD db-id psql' "$MOCK_DOCKER_ARGV_LOG" +grep -Fq 'exec -i -e PGPASSWORD db-id pg_restore' "$MOCK_DOCKER_ARGV_LOG" +[[ "$(grep -Fc 'PGPASSWORD=argv-secret' "$MOCK_DOCKER_ENV_LOG")" == "3" ]] +pass "Docker backup and both restore credential paths inherit PGPASSWORD without argv values" + +! grep -Eq 'capture_command[[:space:]]+paired_backup[[:space:]]+env' "$ROOT_DIR/scripts/run-automatic-backup.sh" +! grep -Eq 'docker exec([^\n]|\\\n)*-e[[:space:]]+PGPASSWORD=' \ + "$ROOT_DIR/scripts/backup-db.sh" "$ROOT_DIR/scripts/restore-db.sh" +pass "production backup scripts contain no value-bearing credential argv patterns" + +find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' -exec rm -rf -- {} + +for day in 01 02 03 04 05 06 07; do + mkdir "$WRAPPER_BACKUPS/hyb-oa-backup-auto-202602${day}T023000Z" +done +rm -f "$WRAPPER_STATE/last-success.json" +expect_failure "off-host finalize rejection fails the source run without publishing success" \ + "${wrapper_env[@]}" MOCK_RSYNC_EXIT=0 MOCK_SSH_EXIT=1 "$ROOT_DIR/scripts/run-automatic-backup.sh" run +[[ ! -e "$WRAPPER_STATE/last-success.json" ]] +[[ "$(find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' | wc -l)" == "8" ]] +[[ -f "$WRAPPER_STATE/last-error.log" && "$(stat -c %a "$WRAPPER_STATE/last-error.log")" == "600" ]] +python3 - "$WRAPPER_STATE/last-run.json" <<'PYTHON' +import json +import sys +with open(sys.argv[1], encoding="utf-8") as handle: + payload = json.load(handle) +assert payload["status"] == "FAIL_AUTOMATIC_BACKUP" +assert payload["findings"][0]["detail"] == "phase_offhost_finalize_failed" +PYTHON +pass "off-host finalize fail-closed preserves the source batch and emits a safe failure category" find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' -exec rm -rf -- {} + for day in 01 02 03 04 05 06 07; do diff --git a/scripts/receive-offhost-backup.sh b/scripts/receive-offhost-backup.sh index c6ce246d..ead2bab3 100755 --- a/scripts/receive-offhost-backup.sh +++ b/scripts/receive-offhost-backup.sh @@ -15,6 +15,8 @@ RRSYNC_BIN="${RRSYNC_BIN:-/usr/bin/rrsync}" PYTHON_BIN="${PYTHON_BIN:-python3}" DAILY_KEEP="${DAILY_KEEP:-7}" WEEKLY_KEEP="${WEEKLY_KEEP:-4}" +declare -A RECONCILED_WEEK_SOURCE=() +declare -a DAILY_ORPHANS_PENDING_DELETE=() fail() { echo "[offhost-receiver][ERROR] $1" >&2 @@ -96,7 +98,8 @@ validate_batch() { [[ "$actual_manifest_sha" == "$expected_manifest_sha" ]] \ || fail "manifest checksum mismatch" - "$PYTHON_BIN" - "$batch_dir" "$run_id" <<'PYTHON' + "$PYTHON_BIN" - "$batch_dir" "$run_id" <<'PYTHON' \ + || fail "batch manifest or artifact validation failed" import hashlib import json import os @@ -135,6 +138,128 @@ for key, expected_name in expected.items(): PYTHON } +validated_batch_identity() { + local batch_dir="$1" + [[ -d "$batch_dir" && ! -L "$batch_dir" ]] || fail "recovery candidate is not a regular directory" + + local run_id manifest_sha identity + run_id="$($PYTHON_BIN - "$batch_dir/manifest.json" <<'PYTHON' +import json +import re +import sys + +with open(sys.argv[1], "r", encoding="utf-8") as handle: + manifest = json.load(handle) +run_id = manifest.get("runId") +if not isinstance(run_id, str) or not re.fullmatch(r"auto-[0-9]{8}T[0-9]{6}Z", run_id): + raise SystemExit("manifest run id is invalid") +print(run_id) +PYTHON + )" || fail "recovery candidate manifest identity is invalid" + manifest_sha="$(sha256sum "$batch_dir/manifest.json" | awk '{print $1}')" + validate_batch "$batch_dir" "$run_id" "$manifest_sha" + identity="$($PYTHON_BIN - "$run_id" <<'PYTHON' +from datetime import datetime +import sys + +run_id = sys.argv[1] +day = run_id.removeprefix("auto-").split("T", 1)[0] +iso = datetime.strptime(day, "%Y%m%d").date().isocalendar() +print(f"{run_id}\t{day}\tweek-{iso.year:04d}-W{iso.week:02d}") +PYTHON + )" || fail "recovery candidate date identity is invalid" + printf '%s\t%s\n' "$identity" "$manifest_sha" +} + +validate_temp_directory() { + local target="$1" + local root="$2" + local pattern="$3" + [[ "$(dirname -- "$target")" == "$root" ]] || fail "orphan target escaped its managed root" + [[ "$(basename -- "$target")" =~ $pattern ]] || fail "orphan target name is invalid" + [[ -d "$target" && ! -L "$target" ]] || fail "orphan target is not a regular directory" +} + +remove_temp_directory() { + local target="$1" + local root="$2" + local pattern="$3" + validate_temp_directory "$target" "$root" "$pattern" + rm -rf -- "$target" +} + +canonical_daily_for_day() { + local run_day="$1" + local -a entries=() + mapfile -t entries < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 \ + -name "hyb-oa-backup-auto-${run_day}T*Z" -print | LC_ALL=C sort) + [[ "${#entries[@]}" -le 1 ]] || fail "daily archive invariant is ambiguous during recovery" + if [[ "${#entries[@]}" -eq 1 ]]; then + printf '%s\n' "${entries[0]}" + fi +} + +reconcile_daily_orphans() { + local -a orphans=() + mapfile -t orphans < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 \ + -name '.daily-replacement-*' -print | LC_ALL=C sort) + local orphan base run_day identity orphan_run_id orphan_day orphan_week orphan_sha + local canonical canonical_identity canonical_run_id canonical_day canonical_week canonical_sha + local -A seen_days=() identities=() canonicals=() + for orphan in "${orphans[@]}"; do + base="$(basename -- "$orphan")" + [[ "$base" =~ ^\.daily-replacement-([0-9]{8})\.([0-9]+)$ ]] \ + || fail "daily replacement orphan name is invalid" + run_day="${BASH_REMATCH[1]}" + [[ -z "${seen_days[$run_day]:-}" ]] || fail "multiple daily replacement orphans exist for one day" + seen_days[$run_day]=1 + validate_temp_directory "$orphan" "$BACKUP_ARCHIVE_DIR" \ + '^\.daily-replacement-[0-9]{8}\.[0-9]+$' + identity="$(validated_batch_identity "$orphan")" \ + || fail "daily replacement orphan validation failed" + IFS=$'\t' read -r orphan_run_id orphan_day orphan_week orphan_sha <<<"$identity" + [[ "$orphan_day" == "$run_day" ]] || fail "daily replacement orphan day does not match its manifest" + identities[$orphan]="$identity" + + canonical="$(canonical_daily_for_day "$run_day")" \ + || fail "daily archive lookup failed during recovery" + canonicals[$run_day]="$canonical" + if [[ -n "$canonical" ]]; then + [[ -d "$canonical" && ! -L "$canonical" ]] || fail "daily archive recovery target is invalid" + canonical_identity="$(validated_batch_identity "$canonical")" \ + || fail "daily archive validation failed during recovery" + IFS=$'\t' read -r canonical_run_id canonical_day canonical_week canonical_sha <<<"$canonical_identity" + [[ "$canonical_day" == "$run_day" \ + && "$(basename -- "$canonical")" == "hyb-oa-backup-$canonical_run_id" ]] \ + || fail "daily archive identity is inconsistent during recovery" + [[ "$canonical_week" == "$orphan_week" ]] || fail "daily replacement week identity is inconsistent" + else + canonical="$BACKUP_ARCHIVE_DIR/hyb-oa-backup-$orphan_run_id" + [[ ! -e "$canonical" ]] || fail "daily recovery target already exists" + fi + done + + for orphan in "${orphans[@]}"; do + base="$(basename -- "$orphan")" + [[ "$base" =~ ^\.daily-replacement-([0-9]{8})\.([0-9]+)$ ]] \ + || fail "daily replacement orphan name changed during recovery" + run_day="${BASH_REMATCH[1]}" + identity="${identities[$orphan]}" + IFS=$'\t' read -r orphan_run_id orphan_day orphan_week orphan_sha <<<"$identity" + canonical="${canonicals[$run_day]}" + if [[ -z "$canonical" ]]; then + canonical="$BACKUP_ARCHIVE_DIR/hyb-oa-backup-$orphan_run_id" + validate_temp_directory "$orphan" "$BACKUP_ARCHIVE_DIR" \ + '^\.daily-replacement-[0-9]{8}\.[0-9]+$' + [[ ! -e "$canonical" ]] || fail "daily recovery target appeared during recovery" + mv -- "$orphan" "$canonical" + else + DAILY_ORPHANS_PENDING_DELETE+=("$orphan") + fi + RECONCILED_WEEK_SOURCE[$orphan_week]="$canonical" + done +} + prune_named_directories() { local root="$1" local prefix="$2" @@ -186,6 +311,138 @@ publish_weekly_checkpoint() { fi } +unique_daily_source_for_week() { + local expected_week="$1" + local -a candidates=() entries=() + mapfile -t entries < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 \ + -name 'hyb-oa-backup-auto-*' -print | LC_ALL=C sort) + local entry identity run_id run_day iso_week manifest_sha + for entry in "${entries[@]}"; do + [[ -d "$entry" && ! -L "$entry" ]] || fail "daily archive candidate is invalid during weekly recovery" + identity="$(validated_batch_identity "$entry")" \ + || fail "daily archive validation failed during weekly recovery" + IFS=$'\t' read -r run_id run_day iso_week manifest_sha <<<"$identity" + [[ "$(basename -- "$entry")" == "hyb-oa-backup-$run_id" ]] \ + || fail "daily archive name does not match its manifest" + if [[ "$iso_week" == "$expected_week" ]]; then + candidates+=("$entry") + fi + done + [[ "${#candidates[@]}" -eq 1 ]] || fail "weekly recovery source is not unique" + printf '%s\n' "${candidates[0]}" +} + +reconcile_weekly_orphans() { + local -a entries=() + mapfile -t entries < <(find "$BACKUP_WEEKLY_DIR" -mindepth 1 -maxdepth 1 \ + \( -name 'week-*-W*.new.*' -o -name 'week-*-W*.rollback.*' \) -print | LC_ALL=C sort) + local -A new_by_week=() rollback_by_week=() weeks=() + local entry base week kind + for entry in "${entries[@]}"; do + base="$(basename -- "$entry")" + if [[ "$base" =~ ^(week-[0-9]{4}-W[0-9]{2})\.(new|rollback)\.([0-9]+)$ ]]; then + week="${BASH_REMATCH[1]}" + kind="${BASH_REMATCH[2]}" + else + fail "weekly orphan name is invalid" + fi + validate_temp_directory "$entry" "$BACKUP_WEEKLY_DIR" \ + '^week-[0-9]{4}-W[0-9]{2}\.(new|rollback)\.[0-9]+$' + weeks[$week]=1 + if [[ "$kind" == "new" ]]; then + [[ -z "${new_by_week[$week]:-}" ]] || fail "multiple weekly staging orphans exist" + new_by_week[$week]="$entry" + else + [[ -z "${rollback_by_week[$week]:-}" ]] || fail "multiple weekly rollback orphans exist" + rollback_by_week[$week]="$entry" + fi + done + for week in "${!RECONCILED_WEEK_SOURCE[@]}"; do + weeks[$week]=1 + done + + local -A sources=() + local canonical new_dir rollback_dir identity run_id run_day iso_week manifest_sha source + for week in "${!weeks[@]}"; do + canonical="$BACKUP_WEEKLY_DIR/$week" + new_dir="${new_by_week[$week]:-}" + rollback_dir="${rollback_by_week[$week]:-}" + + if [[ -e "$canonical" ]]; then + [[ -d "$canonical" && ! -L "$canonical" ]] || fail "weekly checkpoint is not a regular directory" + identity="$(validated_batch_identity "$canonical")" \ + || fail "weekly checkpoint validation failed during recovery" + IFS=$'\t' read -r run_id run_day iso_week manifest_sha <<<"$identity" + [[ "$iso_week" == "$week" ]] || fail "weekly checkpoint manifest belongs to another week" + fi + if [[ -n "$rollback_dir" ]]; then + identity="$(validated_batch_identity "$rollback_dir")" \ + || fail "weekly rollback validation failed" + IFS=$'\t' read -r run_id run_day iso_week manifest_sha <<<"$identity" + [[ "$iso_week" == "$week" ]] || fail "weekly rollback manifest belongs to another week" + fi + + source="${RECONCILED_WEEK_SOURCE[$week]:-}" + if [[ -z "$source" && ! -e "$canonical" && -z "$rollback_dir" && -n "$new_dir" ]]; then + source="$(unique_daily_source_for_week "$week")" \ + || fail "weekly staging recovery source could not be resolved" + fi + if [[ -n "$source" ]]; then + [[ -d "$source" && ! -L "$source" ]] || fail "weekly rebuild source is invalid" + identity="$(validated_batch_identity "$source")" \ + || fail "weekly rebuild source validation failed" + IFS=$'\t' read -r run_id run_day iso_week manifest_sha <<<"$identity" + [[ "$iso_week" == "$week" ]] || fail "weekly rebuild source belongs to another week" + sources[$week]="$source" + fi + [[ -e "$canonical" || -n "$rollback_dir" || -n "$source" ]] \ + || fail "weekly checkpoint has no authoritative recovery source" + done + + for week in "${!weeks[@]}"; do + canonical="$BACKUP_WEEKLY_DIR/$week" + new_dir="${new_by_week[$week]:-}" + rollback_dir="${rollback_by_week[$week]:-}" + + if [[ ! -e "$canonical" && -n "$rollback_dir" ]]; then + validate_temp_directory "$rollback_dir" "$BACKUP_WEEKLY_DIR" \ + '^week-[0-9]{4}-W[0-9]{2}\.rollback\.[0-9]+$' + mv -- "$rollback_dir" "$canonical" + rollback_dir="" + fi + + source="${sources[$week]:-}" + if [[ -n "$source" ]]; then + [[ -d "$source" && ! -L "$source" ]] || fail "weekly rebuild source is invalid" + if ! publish_weekly_checkpoint "$source" "$canonical" "$([[ -e "$canonical" ]] && echo 1 || echo 0)"; then + fail "weekly checkpoint reconciliation failed" + fi + fi + + [[ -e "$canonical" ]] || fail "weekly checkpoint is missing after reconciliation" + if [[ -n "$rollback_dir" ]]; then + remove_temp_directory "$rollback_dir" "$BACKUP_WEEKLY_DIR" \ + '^week-[0-9]{4}-W[0-9]{2}\.rollback\.[0-9]+$' + fi + if [[ -n "$new_dir" ]]; then + remove_temp_directory "$new_dir" "$BACKUP_WEEKLY_DIR" \ + '^week-[0-9]{4}-W[0-9]{2}\.new\.[0-9]+$' + fi + done + local pending_daily_orphan + for pending_daily_orphan in "${DAILY_ORPHANS_PENDING_DELETE[@]}"; do + remove_temp_directory "$pending_daily_orphan" "$BACKUP_ARCHIVE_DIR" \ + '^\.daily-replacement-[0-9]{8}\.[0-9]+$' + done + DAILY_ORPHANS_PENDING_DELETE=() + RECONCILED_WEEK_SOURCE=() +} + +reconcile_interrupted_finalize() { + reconcile_daily_orphans + reconcile_weekly_orphans +} + finalize_batch() { local run_id="$1" local expected_manifest_sha="$2" @@ -194,6 +451,7 @@ finalize_batch() { exec 9>"$BACKUP_ARCHIVE_DIR/.finalize.lock" chmod 0600 "$BACKUP_ARCHIVE_DIR/.finalize.lock" flock -n 9 || fail "another finalize operation is already running" + reconcile_interrupted_finalize local batch_name="hyb-oa-backup-$run_id" local partial_dir="$BACKUP_INBOX_DIR/${batch_name}.partial" diff --git a/scripts/restore-db.sh b/scripts/restore-db.sh index dffee242..d0723197 100755 --- a/scripts/restore-db.sh +++ b/scripts/restore-db.sh @@ -98,7 +98,7 @@ if [[ -z "$POSTGRES_DB" || -z "$POSTGRES_USER" || -z "$POSTGRES_PASSWORD" ]]; th exit 1 fi -ACTUAL_DATABASE="$(docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \ +ACTUAL_DATABASE="$(PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD "$DB_CONTAINER" \ psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atqc 'SELECT current_database()')" if [[ -z "$ACTUAL_DATABASE" || "$ACTUAL_DATABASE" != "$RESTORE_CONFIRM_DATABASE" ]]; then @@ -110,7 +110,7 @@ validate_shadow_database_name "$ACTUAL_DATABASE" docker exec -i "$DB_CONTAINER" pg_restore --list <"$BACKUP_FILE" >/dev/null echo "[INFO] DB restore target confirmed: $ACTUAL_DATABASE (shadow)" -docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \ +PGPASSWORD="$POSTGRES_PASSWORD" docker exec -i -e PGPASSWORD "$DB_CONTAINER" \ pg_restore --clean --if-exists --no-owner --no-acl -U "$POSTGRES_USER" -d "$POSTGRES_DB" \ <"$BACKUP_FILE" diff --git a/scripts/run-automatic-backup.sh b/scripts/run-automatic-backup.sh index b4bc573b..564d63e7 100755 --- a/scripts/run-automatic-backup.sh +++ b/scripts/run-automatic-backup.sh @@ -341,12 +341,15 @@ print(f"{unquote(value.username)}\t{database_name}") [[ "$source_image_digest" =~ ^sha256:[0-9a-f]{64}$ ]] || fail "active server image digest is invalid" CURRENT_PHASE="paired_backup" - if ! capture_command paired_backup env \ - DATABASE_URL="$database_url" DB_TOOL_MODE=docker DB_CONTAINER="$DB_CONTAINER" \ - POSTGRES_DB="$postgres_db" POSTGRES_USER="$postgres_user" POSTGRES_PASSWORD="$postgres_password" \ - STORAGE_DIR_HOST="$STORAGE_HOST_DIR" BACKUP_DIR="$AUTOMATIC_BACKUP_DIR" \ - BACKUP_RUN_ID="$run_id" BUILD_SHA="$BUILD_SHA" SOURCE_IMAGE_DIGEST="$source_image_digest" \ - "$ROOT_DIR/scripts/backup-all.sh"; then + if ! ( + export DATABASE_URL="$database_url" + export DB_TOOL_MODE=docker DB_CONTAINER + export POSTGRES_DB="$postgres_db" POSTGRES_USER="$postgres_user" + export POSTGRES_PASSWORD="$postgres_password" + export STORAGE_DIR_HOST="$STORAGE_HOST_DIR" BACKUP_DIR="$AUTOMATIC_BACKUP_DIR" + export BACKUP_RUN_ID="$run_id" BUILD_SHA SOURCE_IMAGE_DIGEST="$source_image_digest" + capture_command paired_backup "$ROOT_DIR/scripts/backup-all.sh" + ); then fail "paired backup command failed" fi [[ -d "$batch_dir" && -f "$batch_dir/manifest.json" ]] || fail "complete local batch was not published"