From 3c4086555a9e857dbb0db1ebc879cfec3d6f2b74 Mon Sep 17 00:00:00 2001 From: shunli Date: Mon, 27 Jul 2026 07:30:37 +0800 Subject: [PATCH] =?UTF-8?q?=E8=AE=B0=E5=BD=95=E5=BC=82=E6=9C=BA=E5=BD=B1?= =?UTF-8?q?=E5=AD=90=E6=81=A2=E5=A4=8D=E8=AF=81=E6=8D=AE=EF=BC=8807-26-int?= =?UTF-8?q?ernal-ops-recovery-baseline=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .trellis/spec/hyb-oa/deployment-trialrun.md | 34 +++++++++++++++++++ .../implement.md | 7 ++-- .../research/live-read-only-baseline.md | 12 +++++++ 3 files changed, 51 insertions(+), 2 deletions(-) diff --git a/.trellis/spec/hyb-oa/deployment-trialrun.md b/.trellis/spec/hyb-oa/deployment-trialrun.md index 3889c4e8..a17740b0 100644 --- a/.trellis/spec/hyb-oa/deployment-trialrun.md +++ b/.trellis/spec/hyb-oa/deployment-trialrun.md @@ -757,6 +757,13 @@ DB_TOOL_MODE=docker DB_CONTAINER="${DB_CONTAINERS[0]}" \ set of business-table counts with the pre-backup baseline, compares extracted regular-file count with the active storage source, and runs `/ready` using a schema-compatible image. `/ready` alone is not recovery evidence. +- When shadow SQL is supplied through stdin or a heredoc, `docker exec` must use + `-i` and `psql` must use `ON_ERROR_STOP=1`; otherwise an empty result can be + mistaken for a data mismatch even though no verification query ran. +- Compare fixed table counts as a keyed set, not by database-returned row order. + Require the expected unique key count, then normalize both `table=count` + collections with `LC_ALL=C sort` (or an equivalent map comparison) before + equality checking; database collation order is not an evidence contract. ### 4. Validation & Error Matrix @@ -782,6 +789,9 @@ DB_TOOL_MODE=docker DB_CONTAINER="${DB_CONTAINERS[0]}" \ - Shadow migration count, any fixed table count, attachment regular-file count or compatible-image `/ready` differs from baseline -> reject the recovery point and do not enter production cutover. +- Empty, duplicate or missing table-count output -> classify as a verification + execution failure first; do not report production-data drift until stdin was + delivered and every expected key was returned exactly once. ### 5. Good/Base/Bad Cases @@ -791,6 +801,8 @@ DB_TOOL_MODE=docker DB_CONTAINER="${DB_CONTAINERS[0]}" \ after the active application connection is matched to the database container. - Good: a legacy container without `/ready.data.buildSha` records its inspected `sha256:` image ID and leaves `buildSha` null rather than inventing provenance. +- Good: a heredoc query uses `docker exec -i ... psql -v ON_ERROR_STOP=1`, then + compares the complete keyed count set under deterministic `C` sorting. - Base: backup succeeds locally but off-host copy and deployed `/ready` evidence remain explicitly pending; do not claim formal recovery readiness. - Bad: source a stale deployment `.env` and treat a failed dump as a prompt to @@ -808,6 +820,9 @@ DB_TOOL_MODE=docker DB_CONTAINER="${DB_CONTAINERS[0]}" \ SHA-256 values, explicit build-SHA precedence and image-digest preservation. - Run the synthetic drill with Node available and with a `PATH` that excludes Node but includes Python 3; both paths must complete the same round trip. +- For real-format count checks, assert that stdin queries return the full unique + key set, and prove that a different database collation/order does not change + the pass result while a changed count still fails. - Real-instance restore and `/ready` remain separately approved evidence and must never be fabricated from the local drill. @@ -833,3 +848,22 @@ DB_TOOL_MODE=docker DB_CONTAINER= \ SOURCE_IMAGE_DIGEST= ./scripts/backup-all.sh # Real restore remains an approved incident operation after shadow evidence. ``` + +For a containerized count check, the input and comparison contract is: + +```bash +# Wrong: stdin is not attached, and raw row order becomes part of the gate. +actual="$(docker exec shadow-db psql -At -F = <<'SQL' +SELECT table_key, row_count FROM shadow_counts ORDER BY table_key; +SQL +)" +test "$actual" = "$expected" + +# Correct: deliver stdin, fail on SQL errors and compare keyed sets deterministically. +actual="$(docker exec -i shadow-db psql -v ON_ERROR_STOP=1 -At -F = <<'SQL' +SELECT table_key, row_count FROM shadow_counts; +SQL +)" +test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ + "$(printf '%s\n' "$expected" | LC_ALL=C sort)" +``` diff --git a/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.md b/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.md index bc9baca8..6a679e02 100644 --- a/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.md +++ b/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.md @@ -11,8 +11,9 @@ - [x] 运行任务测试计划和 Trellis 治理门禁；按实际结果记录本地 RTO，不伪造 `/ready` 或异机证据。 - [x] 在单独授权后完成真实运行来源、职责分离、异机副本和 `/ready` 的只读核查，并如实记录 `NO_GO` 缺口。 - [x] 经明确授权生成并复制过 DB＋附件＋manifest 批次；后续发现附件来自非活动目录，已将该批次降级为无效恢复证据并保留审计。 -- [ ] 经后续明确授权，从活动 DB 与活动 server 附件挂载源重新生成正确批次并完成异机复验。 -- [ ] 经后续明确授权完成真实职责分配、部署/迁移和真实格式影子恢复后复验。 +- [x] 经后续明确授权，从活动 DB 与活动 server 附件挂载源重新生成正确批次并完成异机复制与目标端复验。 +- [x] 经后续明确授权，使用正确异机批次完成真实备份格式的 DB＋附件隔离影子恢复，并逐项复验迁移数、关键表计数和附件数量。 +- [ ] 经后续明确授权完成真实职责分配与部署/迁移，并在部署环境复验应用 `/ready`、构建标识和数据连续性。 - [x] 修复企微 HR 双 Secret 分工：通讯录 token 只列 userid，自建应用 token 读取详情、部门和 HR 信息。 - [x] 补充单元测试并运行后端语法、聚焦测试、Prisma 与 Trellis 门禁。 - [x] 在真实环境执行两名目标成员的脱敏 dry-run；两人均唯一、不同身份且 active，但批量链路均缺手机和邮箱，已按门禁停止真实写入。 @@ -67,6 +68,8 @@ - 本阶段授权仅覆盖本地文件、测试、规约和提交，不包含生产备份、停机、迁移、部署、timer 安装或企微写入。 - 已确认旧异机批次混用了活动 DB 与非活动附件目录，保留审计但不再作为活动实例成对恢复证据；正确批次须另行批准重做。 - 复核补修后，正式 deploy 的相对/重叠路径负向门禁、显式来源标识、Node 路径以及无 Node/无 Git 的 Python 3 + digest-only 路径均通过；最近两次合成 RTO 均为 7 秒。 +- 已从活动 DB 容器与活动 server 附件挂载生成正确批次并复制到 ARM 备份机；DB catalog、manifest、源/目标产物 SHA-256、附件 tar 与 0 个普通文件计数均通过，且未伪造旧实例 build SHA。ARM 只承担异机备份与恢复演练职责，不运行 OA 服务。 +- 已在 ARM 备份机用不发布端口的一次性 PostgreSQL 16 容器恢复正确批次：58 项成功迁移、12 张关键表计数和 0 个附件普通文件均与冻结基线一致，包含镜像拉取的实测 RTO 为 33 秒。演练后容器、临时附件目录和本轮新增镜像均已删除，未运行 OA server；应用 `/ready` 留待生产部署环境验证。 ## 回滚点 diff --git a/.trellis/tasks/07-26-internal-ops-recovery-baseline/research/live-read-only-baseline.md b/.trellis/tasks/07-26-internal-ops-recovery-baseline/research/live-read-only-baseline.md index e725a1c9..d5f91a29 100644 --- a/.trellis/tasks/07-26-internal-ops-recovery-baseline/research/live-read-only-baseline.md +++ b/.trellis/tasks/07-26-internal-ops-recovery-baseline/research/live-read-only-baseline.md @@ -99,3 +99,15 @@ - 冻结证据必须包含实测旧 Compose project、不可变镜像 ID/digest、成功 migration 数和固定关键表计数；影子恢复必须逐表比对计数、比对附件普通文件数并运行 schema 兼容镜像 `/ready`。 - 当前旧实例没有 build SHA，重做备份不得写入待部署提交。manifest 记录活动容器的实测 `sha256:` 镜像 ID；新版本部署后的备份才能记录其真实 build SHA。 - 后续停机预案采用“旧 PostgreSQL 完全停止后复制到空新目录，旧目录原位保留且不再写入”，不把原子改名后的唯一数据副本当作回滚点。 + +## 2026-07-27 正确异机批次 + +- 已从活动 Compose project 的唯一 DB 容器和活动 server 的真实附件 bind mount 生成正确源端批次 `hyb-oa-backup-20260726T224546Z-live-corrected`；没有读取漂移的部署 `.env` 来猜来源。 +- manifest 状态为 `complete`，来源只记录活动 server 镜像 digest，`buildSha` 保持空值；DB catalog、两份产物 SHA-256、字节数和附件 tar 可读性均已复验。 +- 活动附件目录与归档解出的普通文件数均为 0，与冻结基线及附件相关业务表计数一致；批次目录权限为 `700`。 +- 本轮临时上传的三个备份脚本已精确删除；旧错误批次和历史备份均未删除。 +- 已核对备份机与部署服务器是两个不同目标，补齐本机备份机 SSH 别名，并使用既有密钥认证；未把备份写入部署服务器，也未把备份机当作 OA 部署目标。 +- 正确批次已复制到 ARM 备份机；目标端重新计算 manifest、DB dump 和附件归档 SHA-256，三者均与源端一致，manifest 合同、0 个附件普通文件和目录权限复验通过。旧错误批次继续保留作审计。 +- 已使用正确异机批次在不发布端口的一次性 PostgreSQL 16 容器中完成真实备份格式影子恢复；58 项成功迁移、12 张关键表计数与冻结基线逐项一致，附件归档安全检查和解包后的 0 个普通文件也一致，包含镜像拉取的实测 RTO 为 33 秒。 +- 恢复演练后，一次性数据库容器、临时附件目录和本轮新增 PostgreSQL 镜像均已删除；ARM 继续只承担备份与恢复演练职责，没有运行 OA server 或开放 OA 端口。 +- 当前证据证明 DB＋附件异机恢复点可用，但不替代部署环境的应用 `/ready`、构建标识、Caddy 和前端 marker 验收。生产服务未停止，数据库、附件、容器、Caddy、timer、角色和企微均未修改。