修复(08-08-production-deployment-execution): 持久化远端部署阶段日志
ci-e2e / e2e (push) Canceled after 0s
ci-minimal / check-minimal (push) Canceled after 0s

这个提交包含在:
2026-08-08 21:29:55 +08:00
父节点 b94afcfab0
当前提交 0224ad83d7
共修改 6 个文件,包含 110 行新增和 3 行删除
@@ -1460,6 +1460,9 @@ bash scripts/qa/run-automatic-backup-contract.sh
- Aggregate read-only preflight: `deploy/trialrun/check-existing-deployment.sh`.
- Root-owned forced-command source/installer: `scripts/refresh-deployment-contract.sh` and `deploy/trialrun/install-deployment-contract-refresh.sh`.
- Remote preflight: `deploy/trialrun/preflight-remote.sh` executed over SSH before rsync.
- Persistent long-stage evidence: repo-relative
`artifacts/deployment-runtime/<stage>-<12-char-sha>.log` for `build-init`,
`run-init` and `build-switch-server`.
- Canonical parameter source: allowlisted `Environment=` values from `hyb-oa-automatic-backup-daily.service`.
- Stable image stages: `base`, `client_deps`, `client_build`, `server_deps`, `server_build`.
@@ -1479,6 +1482,20 @@ bash scripts/qa/run-automatic-backup-contract.sh
- After that partial-success state, repair the classified root/program/unit/timer problem, invoke the forced-command `refresh` directly, then rerun aggregate preflight. The application already passed release smoke; do not redeploy it merely to repair the backup build contract, and do not bypass the expected `backup_build` failure while it is stale.
- `apk add` and both lockfile `npm ci` steps precede `BUILD_SHA` / `VITE_BUILD_SHA`. Build identity may invalidate application output, never stable dependency installation.
- Seed and smoke output reports readiness, counts and booleans; it does not print full login identifiers, contact values, WeCom identifiers, redirect URI or generated credentials.
- The three long remote mutation stages write stdout/stderr directly to a
remote file instead of relying on the SSH stream as the only copy. The log
directory is mode `0700`; each fresh log is mode `0600`. Stage names are a
fixed allowlist at call sites, and the filename uses the provenance-verified
12-character candidate prefix.
- A stage success/failure emits only stage, exact child exit code and a
repository-relative `logRef`. Full raw logs remain remote and uncommitted;
after failure, retrieve only the needed tail read-only and redact any
sensitive content before writing task/chat evidence. Do not automatically
dump init/seed logs into ordinary output.
- Logs are created after the verified-source rsync. A later rsync may remove
old runtime artifacts, so retrieve a failed stage log before any approved
retry. Keeping an interactive execution handle is not a substitute for this
second copy.
### 4. Validation & Error Matrix
@@ -1495,16 +1512,28 @@ bash scripts/qa/run-automatic-backup-contract.sh
- Post-deploy installer still embeds the previous build SHA, any managed timer is inactive, or the real backup/watchdog fails runtime identity or off-host freshness -> deployment recovery remains incomplete even when `/ready` is green.
- A later deployment attempt is blocked by `backup_build` after `DEPLOYED_BUT_CONTRACT_REFRESH_FAILED` -> expected fail-closed behavior; repair with direct forced-command `refresh`, not another application release.
- Lockfile unchanged but a new SHA reruns dependency installation -> Dockerfile contract test fails.
- Long stage exits non-zero -> preserve its original status and relative log
reference, stop under `set -e`, and do not enter the next stage.
- SSH stream disappears -> use the already-written remote log for diagnosis;
do not repeat the deployment merely to recreate console output.
- Runtime log directory/file mode is broader than `0700`/`0600`, stage output
has no log reference, or a bare build/init/server-switch call bypasses the
wrapper -> deployment evidence contract fails locally.
### 5. Good/Base/Bad Cases
- Good: wrapper reads the frozen unit contract, preflight passes, Git archive syncs and unchanged dependency layers are cached.
- Good: a long stage writes its complete remote log, returns a stable relative
reference and preserves the exact non-zero child status even if the
interactive stream is incomplete.
- Good: ordinary and forced-command identities pass separately, all aggregate items are reported, deployment passes, root helper updates only fixed build drop-ins, and four timers remain enabled/active.
- Good: a broken-root recovery creates a clean published-commit source tree, proves one environment candidate against running containers without exposing values, copies it with mode `0600`, deploys with `SYNC_ENV=0`, then updates the backup contracts and completes a real off-host backup.
- Base: first initialization uses reviewed explicit parameters and `DEPLOY_DATA_MODE=initialize`; it does not pretend an existing unit is available.
- Bad: inspect a running container's `working_dir`, append another `deploy/trialrun`, then discover the mistake only after rsync.
- Bad: choose one of several stale `.env` files by path or modification time, use `SYNC_ENV=1`, or stop after `/ready` while the backup unit still expects the previous build.
- Bad: grant NOPASSWD to a repository installer, point forced-command at a deploy-user-writable script, reuse an unrestricted root key, stop aggregation after its first failure, or downgrade refresh failure to a warning.
- Bad: run `docker compose build/run/up` as bare SSH-stream-only calls, lose the
terminal tail, then retry production solely to recover diagnostics.
### 6. Tests Required
@@ -1513,6 +1542,9 @@ bash scripts/qa/run-automatic-backup-contract.sh
- `node deploy/trialrun/validate-env-contract.js`.
- `cd server && npm test -- tests/unit/scripts/deployment-safety-contract.test.js`.
- Synthetic fixtures must prove wrong root, missing/symlink env and invalid data fail; static ordering must prove preflight precedes rsync and dependency stages precede build args.
- Static deployment tests must prove all three long stages use the persistent
wrapper, output is redirected to the mode-`0600` file, the error branch
captures `$?` immediately, and the former bare calls are absent.
- A controlled recovery must additionally record: exact published source commit, per-key PASS set/date without values, byte-copy/owner/mode PASS, new-SHA service/timer contract PASS, a successful complete backup, and a successful off-host freshness check. Old candidates remain untouched until a separately approved cleanup.
### 7. Wrong vs Correct
@@ -1538,6 +1570,14 @@ DEPLOY_SERVER=<approved-ssh-target> \
# recovery runbook to prove one complete backup and off-host freshness.
```
```bash
# Wrong: the SSH stream is the only build evidence.
"${compose_cmd[@]}" build init
# Correct: the remote file is authoritative and the stream carries a safe ref.
run_logged_deploy_stage "build-init" "${compose_cmd[@]}" build init
```
## Scenario: Production Migration-First Target Gate
### 1. Scope / Trigger
@@ -80,6 +80,8 @@ bash deploy/trialrun/deploy-existing.sh
包装器固定 `SYNC_ENV=0`、`DEPLOY_DATA_MODE=existing` 和 `DEPLOY_STATUS_OUTPUT_MODE=quiet`。`deploy.sh` 内部的 `prisma migrate deploy` 是本 task 唯一允许的 schema 写入;不允许 seed、业务 API 写入、reset、resolve 或手工 DDL。
远端 `build init`、`run init` 和 `up --build server` 三个长阶段必须把完整 stdout/stderr 写入受控 `artifacts/deployment-runtime/`:目录 `0700`、日志 `0600`,文件名由固定阶段名和已验证候选 12 位 SHA 组成。普通输出只返回 `stage`、真实退出码和仓库相对 `logRef`;SSH 通道中断时已写入内容仍留在远端,后续以只读方式取尾部。原始日志不进 Git/task/普通对话,审查摘要必须脱敏。
部署后 helper 以空 pending allowlist重新执行 target/status/reconciliation,要求 schema up to date;随后才执行 refresh 和只读 readiness。refresh 失败分类为 `DEPLOYED_BUT_CONTRACT_REFRESH_FAILED`,不重复发布应用。
## 6. 回滚窗口
@@ -102,6 +104,7 @@ bash deploy/trialrun/deploy-existing.sh
| 历史身份/refresh/bootstrap | 历史 commit 引用、exact probe 计数、contract/origin 匹配和 DB 引导状态码 | 任一不唯一/不一致即停止,不进入 Prisma status |
| 来源/13 项体检 | commit、published/remoteVerified、13 项 PASS 计数 | 停止,不执行部署写入 |
| target/pre-status/窗口 | 原始受控输出引用、脱敏副本、pending/异常计数、窗口状态 | `PAUSED`,不进入 deploy |
| build/init/server switch | 固定阶段 PASS、相对 `logRef`、候选短 SHA | 远端 `0600` 日志保留,读取尾部后再定性;不得仅凭 SSH 流或盲重试 |
| migrate deploy | deploy wrapper 返回、候选 migration ID | 保留现场,不 reset/resolve/drop/重复部署 |
| post-status/readiness | applied、schema up to date、`/ready`/SHA/smoke | 按窗口状态决定回退或 roll-forward |
| refresh | exact forced-command PASS | `DEPLOYED_BUT_CONTRACT_REFRESH_FAILED`,只修复合同 |
@@ -165,3 +165,17 @@
- termination migration 尚未应用时,终止能力不可能由本轮应用生效;helper 现以 `_prisma_migrations` applied set 走显式分支,不发送 `contract_registry` 状态查询,固定输出 `rollbackWindow=OPEN terminatedRows=0 basis=migration_not_applied`。
- migration 已应用时继续执行真实 `status = 'terminated'` 计数,输出 `basis=terminated_status_count`;查询失败或结果不可解析分别返回固定错误,不使用 catch/default 把未知状态误报为 OPEN。
- 聚焦 fixture 覆盖:未应用且状态查询被设置为必失败仍应 OPEN(证明未查询)、已应用 0 行 OPEN、已应用 2 行 CLOSED,以及已应用查询失败必须 `PROD_TERMINATED_COUNT_UNAVAILABLE`。聚焦测试 19/19 PASS。
## First Production Deployment Attempt Evidence (2026-08-08)
- 窗口依据修复提交 `b94afcf` 已进入 `origin/main`;发布前 preflight 再次通过:精确身份组、13 项聚合体检、数据库三方身份、唯一 pending allowlist 和 `rollbackWindow=OPEN ... basis=migration_not_applied` 全部 PASS。
- 唯一生产入口随后完成已验证源码同步并进入远端 `build init`;编排进程在该长阶段退出,未到 `run init`。代码顺序固定为先 `build init`、后 `prisma:migrate:deploy`,只读复核同时证明 termination migration 仍 pending、公开 `/ready` 仍为旧 build、新 init 镜像仍为旧 build,故 schema 写入和应用切换均为 0。
- 同一候选在本地 `client` typecheck/build PASS;远端当时仍有可用磁盘和内存,未取得明确 OOM 证据,不能把环境猜测写成根因。原裸调用只把输出写入 SSH 流,通道证据不完整,未能恢复真实末尾错误。
- 失败后未重试部署、未运行 reset/resolve/drop、未改生产配置。额外只读检查确认现有 13 项合同仍健康、回滚窗口仍 OPEN;用户随后只批准一次在远端持久化日志机制落地后的受控重试。
## Persistent Remote Stage Log Repair Evidence (2026-08-08)
- `deploy.sh` 为 `build-init`、`run-init` 和 `build-switch-server` 三个长阶段统一增加远端日志包装器;完整 stdout/stderr 直接写入仓库相对 `artifacts/deployment-runtime/`,目录 `0700`、文件 `0600`,文件名只含固定阶段名和已验证候选 12 位 SHA。
- 普通输出仅记录 stage、真实 child exit code 和相对 `logRef`。失败状态在 Bash `else` 分支中立即捕获并返回,随后由 `set -e` 停止,不会因 `fi` 后读取 `$?` 而丢失原码。
- 原始 build/init 日志不写入 Git、task 或普通对话;若受控重试失败,先只读取对应远端日志尾部并脱敏定性,再决定后续动作。任何下一次 rsync 可能清除旧 runtime artifacts,因此读取必须发生在重试之前。
- 聚焦静态回归证明三个裸调用均已替换、重定向和权限合同存在、失败分支立即捕获退出码;shell syntax、env contract 与部署安全测试 PASS。
+29 -3
查看文件
@@ -263,6 +263,32 @@ if [[ "$ENABLE_WORKER" == "1" ]]; then
profile_args+=(--profile worker)
fi
DEPLOY_RUNTIME_LOG_REF="artifacts/deployment-runtime"
DEPLOY_RUNTIME_LOG_DIR="$REMOTE_DIR/$DEPLOY_RUNTIME_LOG_REF"
install -d -m 0700 "$DEPLOY_RUNTIME_LOG_DIR"
run_logged_deploy_stage() {
local stage="$1"
shift
[[ "$stage" =~ ^[a-z0-9-]+$ ]] || {
echo "[deploy-stage][ERROR] 阶段名称格式无效" >&2
return 2
}
local log_ref="$DEPLOY_RUNTIME_LOG_REF/${stage}-${BUILD_SHA:0:12}.log"
local log_file="$REMOTE_DIR/$log_ref"
: > "$log_file"
chmod 0600 "$log_file"
echo "[deploy-stage] stage=$stage logRef=$log_ref"
if "$@" </dev/null >"$log_file" 2>&1; then
echo "[deploy-stage][PASS] stage=$stage logRef=$log_ref"
return 0
else
local status=$?
echo "[deploy-stage][ERROR] stage=$stage exit=$status logRef=$log_ref" >&2
return "$status"
fi
}
echo ">>> 4.1 校验远端 compose 配置"
"${compose_cmd[@]}" "${profile_args[@]}" config >/dev/null
@@ -273,13 +299,13 @@ if [[ "$ENABLE_MINIO" == "1" ]]; then
fi
echo ">>> 4.3 构建最新 init 镜像"
"${compose_cmd[@]}" build init
run_logged_deploy_stage "build-init" "${compose_cmd[@]}" build init
echo ">>> 4.4 预检新配置并执行 init"
"${compose_cmd[@]}" run --rm -T --entrypoint sh init -lc 'cd /app/server && node -e '\''require("./src/config").getConfig()'\'' && npm run prisma:migrate:deploy && npm run seed:config && npm run seed:trialrun' </dev/null
run_logged_deploy_stage "run-init" "${compose_cmd[@]}" run --rm -T --entrypoint sh init -lc 'cd /app/server && node -e '\''require("./src/config").getConfig()'\'' && npm run prisma:migrate:deploy && npm run seed:config && npm run seed:trialrun'
echo ">>> 4.5 构建并切换应用服务"
"${compose_cmd[@]}" "${profile_args[@]}" up -d --build server
run_logged_deploy_stage "build-switch-server" "${compose_cmd[@]}" "${profile_args[@]}" up -d --build server
if [[ "$ENABLE_WORKER" == "1" ]]; then
"${compose_cmd[@]}" "${profile_args[@]}" up -d worker
fi
+13
查看文件
@@ -746,6 +746,19 @@ if (deployExistingScriptContent.indexOf('ssh "$DEPLOY_REFRESH_SERVER" refresh')
errors.push('deploy-existing.sh 必须在应用部署成功后刷新备份合同');
}
for (const requiredFragment of [
'DEPLOY_RUNTIME_LOG_REF="artifacts/deployment-runtime"',
'install -d -m 0700 "$DEPLOY_RUNTIME_LOG_DIR"',
'chmod 0600 "$log_file"',
'run_logged_deploy_stage "build-init"',
'run_logged_deploy_stage "run-init"',
'run_logged_deploy_stage "build-switch-server"',
]) {
if (!deployScriptContent.includes(requiredFragment)) {
errors.push(`deploy.sh 缺少远端阶段日志合同: ${requiredFragment}`);
}
}
for (const requiredFragment of [
'validateProductionOrigin',
'validateProductionDbName',
@@ -96,6 +96,17 @@ describe("deployment safety contract", () => {
expect(wrapper).toContain("DEPLOY_STATUS_OUTPUT_MODE=quiet");
expect(deploy).toContain("DEPLOY_STATUS_OUTPUT_MODE");
expect(deploy).toContain("DEPLOY_STATUS_OUTPUT_MODE='${DEPLOY_STATUS_OUTPUT_MODE}'");
expect(deploy).toContain('DEPLOY_RUNTIME_LOG_REF="artifacts/deployment-runtime"');
expect(deploy).toContain('install -d -m 0700 "$DEPLOY_RUNTIME_LOG_DIR"');
expect(deploy).toContain('chmod 0600 "$log_file"');
expect(deploy).toContain('>"$log_file" 2>&1');
expect(deploy).toMatch(/if "\$@" <\/dev\/null >"\$log_file" 2>&1; then[\s\S]*else\s+local status=\$\?/u);
expect(deploy).toContain('run_logged_deploy_stage "build-init"');
expect(deploy).toContain('run_logged_deploy_stage "run-init"');
expect(deploy).toContain('run_logged_deploy_stage "build-switch-server"');
expect(deploy).not.toContain('\n"${compose_cmd[@]}" build init\n');
expect(deploy).not.toContain('\n"${compose_cmd[@]}" run --rm -T --entrypoint sh init');
expect(deploy).not.toContain('\n"${compose_cmd[@]}" "${profile_args[@]}" up -d --build server\n');
expect(wrapper).toContain("生产部署验证不接受未进入上游的提交");
expect(wrapper).toContain("部署后 migration status 与目标只读复核");
expect(wrapper).toContain('EXPECTED_PENDING_MIGRATIONS=""');