diff --git a/.trellis/spec/hyb-oa/deployment-trialrun.md b/.trellis/spec/hyb-oa/deployment-trialrun.md index d0c90d7a..77add8b4 100644 --- a/.trellis/spec/hyb-oa/deployment-trialrun.md +++ b/.trellis/spec/hyb-oa/deployment-trialrun.md @@ -1460,6 +1460,9 @@ bash scripts/qa/run-automatic-backup-contract.sh - Aggregate read-only preflight: `deploy/trialrun/check-existing-deployment.sh`. - Root-owned forced-command source/installer: `scripts/refresh-deployment-contract.sh` and `deploy/trialrun/install-deployment-contract-refresh.sh`. - Remote preflight: `deploy/trialrun/preflight-remote.sh` executed over SSH before rsync. +- Persistent long-stage evidence: repo-relative + `artifacts/deployment-runtime/-<12-char-sha>.log` for `build-init`, + `run-init` and `build-switch-server`. - Canonical parameter source: allowlisted `Environment=` values from `hyb-oa-automatic-backup-daily.service`. - Stable image stages: `base`, `client_deps`, `client_build`, `server_deps`, `server_build`. @@ -1479,6 +1482,20 @@ bash scripts/qa/run-automatic-backup-contract.sh - After that partial-success state, repair the classified root/program/unit/timer problem, invoke the forced-command `refresh` directly, then rerun aggregate preflight. The application already passed release smoke; do not redeploy it merely to repair the backup build contract, and do not bypass the expected `backup_build` failure while it is stale. - `apk add` and both lockfile `npm ci` steps precede `BUILD_SHA` / `VITE_BUILD_SHA`. Build identity may invalidate application output, never stable dependency installation. - Seed and smoke output reports readiness, counts and booleans; it does not print full login identifiers, contact values, WeCom identifiers, redirect URI or generated credentials. +- The three long remote mutation stages write stdout/stderr directly to a + remote file instead of relying on the SSH stream as the only copy. The log + directory is mode `0700`; each fresh log is mode `0600`. Stage names are a + fixed allowlist at call sites, and the filename uses the provenance-verified + 12-character candidate prefix. +- A stage success/failure emits only stage, exact child exit code and a + repository-relative `logRef`. Full raw logs remain remote and uncommitted; + after failure, retrieve only the needed tail read-only and redact any + sensitive content before writing task/chat evidence. Do not automatically + dump init/seed logs into ordinary output. +- Logs are created after the verified-source rsync. A later rsync may remove + old runtime artifacts, so retrieve a failed stage log before any approved + retry. Keeping an interactive execution handle is not a substitute for this + second copy. ### 4. Validation & Error Matrix @@ -1495,16 +1512,28 @@ bash scripts/qa/run-automatic-backup-contract.sh - Post-deploy installer still embeds the previous build SHA, any managed timer is inactive, or the real backup/watchdog fails runtime identity or off-host freshness -> deployment recovery remains incomplete even when `/ready` is green. - A later deployment attempt is blocked by `backup_build` after `DEPLOYED_BUT_CONTRACT_REFRESH_FAILED` -> expected fail-closed behavior; repair with direct forced-command `refresh`, not another application release. - Lockfile unchanged but a new SHA reruns dependency installation -> Dockerfile contract test fails. +- Long stage exits non-zero -> preserve its original status and relative log + reference, stop under `set -e`, and do not enter the next stage. +- SSH stream disappears -> use the already-written remote log for diagnosis; + do not repeat the deployment merely to recreate console output. +- Runtime log directory/file mode is broader than `0700`/`0600`, stage output + has no log reference, or a bare build/init/server-switch call bypasses the + wrapper -> deployment evidence contract fails locally. ### 5. Good/Base/Bad Cases - Good: wrapper reads the frozen unit contract, preflight passes, Git archive syncs and unchanged dependency layers are cached. +- Good: a long stage writes its complete remote log, returns a stable relative + reference and preserves the exact non-zero child status even if the + interactive stream is incomplete. - Good: ordinary and forced-command identities pass separately, all aggregate items are reported, deployment passes, root helper updates only fixed build drop-ins, and four timers remain enabled/active. - Good: a broken-root recovery creates a clean published-commit source tree, proves one environment candidate against running containers without exposing values, copies it with mode `0600`, deploys with `SYNC_ENV=0`, then updates the backup contracts and completes a real off-host backup. - Base: first initialization uses reviewed explicit parameters and `DEPLOY_DATA_MODE=initialize`; it does not pretend an existing unit is available. - Bad: inspect a running container's `working_dir`, append another `deploy/trialrun`, then discover the mistake only after rsync. - Bad: choose one of several stale `.env` files by path or modification time, use `SYNC_ENV=1`, or stop after `/ready` while the backup unit still expects the previous build. - Bad: grant NOPASSWD to a repository installer, point forced-command at a deploy-user-writable script, reuse an unrestricted root key, stop aggregation after its first failure, or downgrade refresh failure to a warning. +- Bad: run `docker compose build/run/up` as bare SSH-stream-only calls, lose the + terminal tail, then retry production solely to recover diagnostics. ### 6. Tests Required @@ -1513,6 +1542,9 @@ bash scripts/qa/run-automatic-backup-contract.sh - `node deploy/trialrun/validate-env-contract.js`. - `cd server && npm test -- tests/unit/scripts/deployment-safety-contract.test.js`. - Synthetic fixtures must prove wrong root, missing/symlink env and invalid data fail; static ordering must prove preflight precedes rsync and dependency stages precede build args. +- Static deployment tests must prove all three long stages use the persistent + wrapper, output is redirected to the mode-`0600` file, the error branch + captures `$?` immediately, and the former bare calls are absent. - A controlled recovery must additionally record: exact published source commit, per-key PASS set/date without values, byte-copy/owner/mode PASS, new-SHA service/timer contract PASS, a successful complete backup, and a successful off-host freshness check. Old candidates remain untouched until a separately approved cleanup. ### 7. Wrong vs Correct @@ -1538,6 +1570,14 @@ DEPLOY_SERVER= \ # recovery runbook to prove one complete backup and off-host freshness. ``` +```bash +# Wrong: the SSH stream is the only build evidence. +"${compose_cmd[@]}" build init + +# Correct: the remote file is authoritative and the stream carries a safe ref. +run_logged_deploy_stage "build-init" "${compose_cmd[@]}" build init +``` + ## Scenario: Production Migration-First Target Gate ### 1. Scope / Trigger diff --git a/.trellis/tasks/08-08-production-deployment-execution/design.md b/.trellis/tasks/08-08-production-deployment-execution/design.md index d800a3b9..4730a798 100644 --- a/.trellis/tasks/08-08-production-deployment-execution/design.md +++ b/.trellis/tasks/08-08-production-deployment-execution/design.md @@ -80,6 +80,8 @@ bash deploy/trialrun/deploy-existing.sh 包装器固定 `SYNC_ENV=0`、`DEPLOY_DATA_MODE=existing` 和 `DEPLOY_STATUS_OUTPUT_MODE=quiet`。`deploy.sh` 内部的 `prisma migrate deploy` 是本 task 唯一允许的 schema 写入;不允许 seed、业务 API 写入、reset、resolve 或手工 DDL。 +远端 `build init`、`run init` 和 `up --build server` 三个长阶段必须把完整 stdout/stderr 写入受控 `artifacts/deployment-runtime/`:目录 `0700`、日志 `0600`,文件名由固定阶段名和已验证候选 12 位 SHA 组成。普通输出只返回 `stage`、真实退出码和仓库相对 `logRef`;SSH 通道中断时已写入内容仍留在远端,后续以只读方式取尾部。原始日志不进 Git/task/普通对话,审查摘要必须脱敏。 + 部署后 helper 以空 pending allowlist重新执行 target/status/reconciliation,要求 schema up to date;随后才执行 refresh 和只读 readiness。refresh 失败分类为 `DEPLOYED_BUT_CONTRACT_REFRESH_FAILED`,不重复发布应用。 ## 6. 回滚窗口 @@ -102,6 +104,7 @@ bash deploy/trialrun/deploy-existing.sh | 历史身份/refresh/bootstrap | 历史 commit 引用、exact probe 计数、contract/origin 匹配和 DB 引导状态码 | 任一不唯一/不一致即停止,不进入 Prisma status | | 来源/13 项体检 | commit、published/remoteVerified、13 项 PASS 计数 | 停止,不执行部署写入 | | target/pre-status/窗口 | 原始受控输出引用、脱敏副本、pending/异常计数、窗口状态 | `PAUSED`,不进入 deploy | +| build/init/server switch | 固定阶段 PASS、相对 `logRef`、候选短 SHA | 远端 `0600` 日志保留,读取尾部后再定性;不得仅凭 SSH 流或盲重试 | | migrate deploy | deploy wrapper 返回、候选 migration ID | 保留现场,不 reset/resolve/drop/重复部署 | | post-status/readiness | applied、schema up to date、`/ready`/SHA/smoke | 按窗口状态决定回退或 roll-forward | | refresh | exact forced-command PASS | `DEPLOYED_BUT_CONTRACT_REFRESH_FAILED`,只修复合同 | diff --git a/.trellis/tasks/08-08-production-deployment-execution/implement.md b/.trellis/tasks/08-08-production-deployment-execution/implement.md index 2b4087df..83b1cf6c 100644 --- a/.trellis/tasks/08-08-production-deployment-execution/implement.md +++ b/.trellis/tasks/08-08-production-deployment-execution/implement.md @@ -165,3 +165,17 @@ - termination migration 尚未应用时,终止能力不可能由本轮应用生效;helper 现以 `_prisma_migrations` applied set 走显式分支,不发送 `contract_registry` 状态查询,固定输出 `rollbackWindow=OPEN terminatedRows=0 basis=migration_not_applied`。 - migration 已应用时继续执行真实 `status = 'terminated'` 计数,输出 `basis=terminated_status_count`;查询失败或结果不可解析分别返回固定错误,不使用 catch/default 把未知状态误报为 OPEN。 - 聚焦 fixture 覆盖:未应用且状态查询被设置为必失败仍应 OPEN(证明未查询)、已应用 0 行 OPEN、已应用 2 行 CLOSED,以及已应用查询失败必须 `PROD_TERMINATED_COUNT_UNAVAILABLE`。聚焦测试 19/19 PASS。 + +## First Production Deployment Attempt Evidence (2026-08-08) + +- 窗口依据修复提交 `b94afcf` 已进入 `origin/main`;发布前 preflight 再次通过:精确身份组、13 项聚合体检、数据库三方身份、唯一 pending allowlist 和 `rollbackWindow=OPEN ... basis=migration_not_applied` 全部 PASS。 +- 唯一生产入口随后完成已验证源码同步并进入远端 `build init`;编排进程在该长阶段退出,未到 `run init`。代码顺序固定为先 `build init`、后 `prisma:migrate:deploy`,只读复核同时证明 termination migration 仍 pending、公开 `/ready` 仍为旧 build、新 init 镜像仍为旧 build,故 schema 写入和应用切换均为 0。 +- 同一候选在本地 `client` typecheck/build PASS;远端当时仍有可用磁盘和内存,未取得明确 OOM 证据,不能把环境猜测写成根因。原裸调用只把输出写入 SSH 流,通道证据不完整,未能恢复真实末尾错误。 +- 失败后未重试部署、未运行 reset/resolve/drop、未改生产配置。额外只读检查确认现有 13 项合同仍健康、回滚窗口仍 OPEN;用户随后只批准一次在远端持久化日志机制落地后的受控重试。 + +## Persistent Remote Stage Log Repair Evidence (2026-08-08) + +- `deploy.sh` 为 `build-init`、`run-init` 和 `build-switch-server` 三个长阶段统一增加远端日志包装器;完整 stdout/stderr 直接写入仓库相对 `artifacts/deployment-runtime/`,目录 `0700`、文件 `0600`,文件名只含固定阶段名和已验证候选 12 位 SHA。 +- 普通输出仅记录 stage、真实 child exit code 和相对 `logRef`。失败状态在 Bash `else` 分支中立即捕获并返回,随后由 `set -e` 停止,不会因 `fi` 后读取 `$?` 而丢失原码。 +- 原始 build/init 日志不写入 Git、task 或普通对话;若受控重试失败,先只读取对应远端日志尾部并脱敏定性,再决定后续动作。任何下一次 rsync 可能清除旧 runtime artifacts,因此读取必须发生在重试之前。 +- 聚焦静态回归证明三个裸调用均已替换、重定向和权限合同存在、失败分支立即捕获退出码;shell syntax、env contract 与部署安全测试 PASS。 diff --git a/deploy/trialrun/deploy.sh b/deploy/trialrun/deploy.sh index 074a32e0..1ffefb06 100755 --- a/deploy/trialrun/deploy.sh +++ b/deploy/trialrun/deploy.sh @@ -263,6 +263,32 @@ if [[ "$ENABLE_WORKER" == "1" ]]; then profile_args+=(--profile worker) fi +DEPLOY_RUNTIME_LOG_REF="artifacts/deployment-runtime" +DEPLOY_RUNTIME_LOG_DIR="$REMOTE_DIR/$DEPLOY_RUNTIME_LOG_REF" +install -d -m 0700 "$DEPLOY_RUNTIME_LOG_DIR" + +run_logged_deploy_stage() { + local stage="$1" + shift + [[ "$stage" =~ ^[a-z0-9-]+$ ]] || { + echo "[deploy-stage][ERROR] 阶段名称格式无效" >&2 + return 2 + } + local log_ref="$DEPLOY_RUNTIME_LOG_REF/${stage}-${BUILD_SHA:0:12}.log" + local log_file="$REMOTE_DIR/$log_ref" + : > "$log_file" + chmod 0600 "$log_file" + echo "[deploy-stage] stage=$stage logRef=$log_ref" + if "$@" "$log_file" 2>&1; then + echo "[deploy-stage][PASS] stage=$stage logRef=$log_ref" + return 0 + else + local status=$? + echo "[deploy-stage][ERROR] stage=$stage exit=$status logRef=$log_ref" >&2 + return "$status" + fi +} + echo ">>> 4.1 校验远端 compose 配置" "${compose_cmd[@]}" "${profile_args[@]}" config >/dev/null @@ -273,13 +299,13 @@ if [[ "$ENABLE_MINIO" == "1" ]]; then fi echo ">>> 4.3 构建最新 init 镜像" -"${compose_cmd[@]}" build init +run_logged_deploy_stage "build-init" "${compose_cmd[@]}" build init echo ">>> 4.4 预检新配置并执行 init" -"${compose_cmd[@]}" run --rm -T --entrypoint sh init -lc 'cd /app/server && node -e '\''require("./src/config").getConfig()'\'' && npm run prisma:migrate:deploy && npm run seed:config && npm run seed:trialrun' >> 4.5 构建并切换应用服务" -"${compose_cmd[@]}" "${profile_args[@]}" up -d --build server +run_logged_deploy_stage "build-switch-server" "${compose_cmd[@]}" "${profile_args[@]}" up -d --build server if [[ "$ENABLE_WORKER" == "1" ]]; then "${compose_cmd[@]}" "${profile_args[@]}" up -d worker fi diff --git a/deploy/trialrun/validate-env-contract.js b/deploy/trialrun/validate-env-contract.js index f9c957c8..0c117994 100755 --- a/deploy/trialrun/validate-env-contract.js +++ b/deploy/trialrun/validate-env-contract.js @@ -746,6 +746,19 @@ if (deployExistingScriptContent.indexOf('ssh "$DEPLOY_REFRESH_SERVER" refresh') errors.push('deploy-existing.sh 必须在应用部署成功后刷新备份合同'); } +for (const requiredFragment of [ + 'DEPLOY_RUNTIME_LOG_REF="artifacts/deployment-runtime"', + 'install -d -m 0700 "$DEPLOY_RUNTIME_LOG_DIR"', + 'chmod 0600 "$log_file"', + 'run_logged_deploy_stage "build-init"', + 'run_logged_deploy_stage "run-init"', + 'run_logged_deploy_stage "build-switch-server"', +]) { + if (!deployScriptContent.includes(requiredFragment)) { + errors.push(`deploy.sh 缺少远端阶段日志合同: ${requiredFragment}`); + } +} + for (const requiredFragment of [ 'validateProductionOrigin', 'validateProductionDbName', diff --git a/server/tests/unit/scripts/deployment-safety-contract.test.js b/server/tests/unit/scripts/deployment-safety-contract.test.js index bcd5cdef..6a72ef66 100644 --- a/server/tests/unit/scripts/deployment-safety-contract.test.js +++ b/server/tests/unit/scripts/deployment-safety-contract.test.js @@ -96,6 +96,17 @@ describe("deployment safety contract", () => { expect(wrapper).toContain("DEPLOY_STATUS_OUTPUT_MODE=quiet"); expect(deploy).toContain("DEPLOY_STATUS_OUTPUT_MODE"); expect(deploy).toContain("DEPLOY_STATUS_OUTPUT_MODE='${DEPLOY_STATUS_OUTPUT_MODE}'"); + expect(deploy).toContain('DEPLOY_RUNTIME_LOG_REF="artifacts/deployment-runtime"'); + expect(deploy).toContain('install -d -m 0700 "$DEPLOY_RUNTIME_LOG_DIR"'); + expect(deploy).toContain('chmod 0600 "$log_file"'); + expect(deploy).toContain('>"$log_file" 2>&1'); + expect(deploy).toMatch(/if "\$@" <\/dev\/null >"\$log_file" 2>&1; then[\s\S]*else\s+local status=\$\?/u); + expect(deploy).toContain('run_logged_deploy_stage "build-init"'); + expect(deploy).toContain('run_logged_deploy_stage "run-init"'); + expect(deploy).toContain('run_logged_deploy_stage "build-switch-server"'); + expect(deploy).not.toContain('\n"${compose_cmd[@]}" build init\n'); + expect(deploy).not.toContain('\n"${compose_cmd[@]}" run --rm -T --entrypoint sh init'); + expect(deploy).not.toContain('\n"${compose_cmd[@]}" "${profile_args[@]}" up -d --build server\n'); expect(wrapper).toContain("生产部署验证不接受未进入上游的提交"); expect(wrapper).toContain("部署后 migration status 与目标只读复核"); expect(wrapper).toContain('EXPECTED_PENDING_MIGRATIONS=""');