比较提交
11
次代码提交
| 作者 | SHA1 | 提交日期 | |
|---|---|---|---|
 wangsli
|
cf63d3cb1b | ||
|
wangsli
|
21c49dd941 | ||
|
wangsli
|
01f1259557 | ||
|
wangsli
|
f1e8cf138a | ||
|
wangsli
|
95b51e6245 | ||
|
wangsli
|
de35ab8cf6 | ||
|
wangsli
|
a0006f07b8 | ||
|
wangsli
|
2526052e73 | ||
|
wangsli
|
9875bc37a4 | ||
|
wangsli
|
b78f2a11ce | ||
|
wangsli
|
325c592aba |
@@ -619,6 +619,9 @@ curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"
|
||||
- Application storage mount: `${STORAGE_HOST_DIR:-./data/storage}`.
|
||||
- Formal deploy inputs: absolute `POSTGRES_DATA_DIR`, absolute
|
||||
`STORAGE_HOST_DIR`, `DEPLOY_DATA_MODE=existing|initialize`.
|
||||
- Timer install inputs: `COMPOSE_PROJECT_NAME`, absolute
|
||||
`POSTGRES_DATA_DIR`, absolute `STORAGE_HOST_DIR`, full 40–64 hex
|
||||
`BUILD_SHA` and its exact 12-hex `VITE_BUILD_SHA` prefix.
|
||||
- DB has no host `ports`; server remains loopback-only.
|
||||
|
||||
### 3. Contracts
|
||||
@@ -639,6 +642,16 @@ curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"
|
||||
only that mode may create data directories.
|
||||
- Remote readiness may use host curl, but build-SHA JSON validation runs with
|
||||
Node inside the server container; the Docker host does not need Node.
|
||||
- The timer installer must validate the same non-root, non-overlapping absolute
|
||||
data-path shape as formal deploy, require an existing `PG_VERSION` and
|
||||
storage directory, and write both paths plus both build identifiers into the
|
||||
systemd service `Environment=` entries. Project name alone is insufficient:
|
||||
a later timer invocation would otherwise interpolate Compose defaults and
|
||||
operate against a non-active data pair.
|
||||
- Timer installation may enable/start the timer unit, but verification must not
|
||||
manually start the daily service because that path can send real alerts.
|
||||
Verify the unit environment against active container mounts, then run
|
||||
Compose `config --quiet` and `exec -T server true` with the same values.
|
||||
- For an existing cutover, stop the old PostgreSQL completely, copy PGDATA and
|
||||
storage into pre-verified empty absolute targets while preserving ownership,
|
||||
permissions, links and extended attributes, and retain the old directories
|
||||
@@ -651,6 +664,13 @@ curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"
|
||||
- Any fixed `container_name` appears -> environment contract fails.
|
||||
- Required data mount loses its parameter -> environment contract fails.
|
||||
- Deploy/timer omits `-p` -> environment contract fails.
|
||||
- Timer install omits either data path/build identifier, receives nested paths,
|
||||
a PGDATA without `PG_VERSION`, a missing storage directory, an invalid full
|
||||
SHA or a 12-character value that is not its prefix -> fail before writing the
|
||||
systemd unit.
|
||||
- A path-segment regex can match an empty string at the leading `/` -> contract
|
||||
failure; use explicit root, trailing slash, `//`, `/./` and `/../` checks so
|
||||
valid absolute paths remain installable.
|
||||
- Formal deploy omits paths/mode, uses a relative/broad/overlapping path, or an
|
||||
`existing` PGDATA lacks `PG_VERSION` -> fail before DB start.
|
||||
- New project resolves the same PGDATA while the old DB runs -> stop before
|
||||
@@ -664,17 +684,26 @@ curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"
|
||||
- The isolated backup/restore drill mutates Compose with a DB port and fixed
|
||||
name and proves the contract rejects it.
|
||||
- `bash -n` covers deploy, timer, backup and restore scripts.
|
||||
- Timer contract assertions cover all four explicit inputs, path isolation,
|
||||
PGDATA marker, SHA/prefix validation and the four systemd `Environment=`
|
||||
entries. On the deployment host, assert timer enabled/active, daily service
|
||||
inactive, exact active-environment equality and non-mutating Compose resolve.
|
||||
|
||||
### 6. Good/Base/Bad Cases
|
||||
|
||||
- Good: one approved project resolves one stopped-and-migrated PGDATA and the
|
||||
matching active storage path; deploy, timer and backup all use that identity.
|
||||
- Good: the timer unit contains the current project, active mount sources and
|
||||
deployed full/short SHA pair; Compose config and a non-mutating server exec
|
||||
resolve under those exact values while the daily service stays inactive.
|
||||
- Base: direct local Compose uses the documented relative defaults under one
|
||||
project; those defaults are never accepted by formal `deploy.sh`.
|
||||
- Bad: start a renamed project before the old DB stops, or let it create a new
|
||||
empty directory and accept a green health check as proof of data continuity.
|
||||
- Bad: move the only PGDATA into the new path, migrate it in place, then assume
|
||||
the old container name alone is a rollback point.
|
||||
- Bad: reinstall the timer with only `COMPOSE_PROJECT_NAME`, allowing systemd
|
||||
Compose interpolation to fall back to repository-relative data directories.
|
||||
|
||||
### 7. Wrong vs Correct
|
||||
|
||||
@@ -697,6 +726,22 @@ DB_TOOL_MODE=docker DB_CONTAINER="${DB_CONTAINERS[0]}" \
|
||||
STORAGE_DIR_HOST=<verified-active-storage> ./scripts/backup-all.sh
|
||||
```
|
||||
|
||||
For timer installation, keep the active deployment identity explicit:
|
||||
|
||||
```bash
|
||||
# Wrong: project identity alone still permits relative data-path fallback.
|
||||
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
|
||||
bash deploy/trialrun/install-warch3-ops-timer.sh install
|
||||
|
||||
# Correct: use the verified active mounts and deployed SHA pair.
|
||||
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
|
||||
POSTGRES_DATA_DIR=<verified-active-pgdata> \
|
||||
STORAGE_HOST_DIR=<verified-active-storage> \
|
||||
BUILD_SHA=<verified-full-build-sha> \
|
||||
VITE_BUILD_SHA=<verified-12-char-prefix> \
|
||||
bash deploy/trialrun/install-warch3-ops-timer.sh install
|
||||
```
|
||||
|
||||
## Scenario: Paired Backup And Shadow-Only Restore
|
||||
|
||||
### 1. Scope / Trigger
|
||||
@@ -867,3 +912,188 @@ SQL
|
||||
test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
|
||||
"$(printf '%s\n' "$expected" | LC_ALL=C sort)"
|
||||
```
|
||||
|
||||
## Scenario: Daily Restricted Off-Host Backup
|
||||
|
||||
### 1. Scope / Trigger
|
||||
|
||||
- Trigger: installing or changing automatic paired backups, off-host transfer,
|
||||
retention, freshness alerts or quarterly restore reminders.
|
||||
- Purpose: keep a 24-hour recovery point without giving the production host
|
||||
ordinary shell or delete access to the ARM archive.
|
||||
|
||||
### 2. Signatures
|
||||
|
||||
- Production runner: `scripts/run-automatic-backup.sh` with `run`,
|
||||
`check-stale` and `remind-drill` modes.
|
||||
- ARM forced receiver: `scripts/receive-offhost-backup.sh`.
|
||||
- Shared host alert helper: `scripts/send-wecom-group-alert.py` with explicit
|
||||
`--webhook-file`, `--summary-file` and `--state-file` paths.
|
||||
- ARM watchdog: `scripts/check-offhost-backup-deadman.py` with archive root,
|
||||
state root, alert helper, Webhook file and a default `93600` second threshold.
|
||||
- Installers: `install-automatic-backup-timer.sh` and
|
||||
`install-offhost-backup-receiver.sh`.
|
||||
- Local contract: `bash scripts/qa/run-automatic-backup-contract.sh`.
|
||||
- Production required environment: `REMOTE_DIR`, `REMOTE_ENV_FILE`,
|
||||
`COMPOSE_PROJECT_NAME`, `POSTGRES_DATA_DIR`, `STORAGE_HOST_DIR`, `BUILD_SHA`,
|
||||
`VITE_BUILD_SHA`, `AUTOMATIC_BACKUP_DIR`, `AUTOMATIC_BACKUP_STATE_DIR`,
|
||||
`OFFHOST_TARGET`, `OFFHOST_SSH_KEY`, `OFFHOST_KNOWN_HOSTS`,
|
||||
`ALERT_WEBHOOK_FILE` and `ALERT_HELPER`.
|
||||
- Receiver config: independent absolute `BACKUP_INBOX_DIR`,
|
||||
`BACKUP_ARCHIVE_DIR`, `BACKUP_WEEKLY_DIR`, `RRSYNC_BIN`, `PYTHON_BIN`,
|
||||
`DAILY_KEEP` and `WEEKLY_KEEP`.
|
||||
|
||||
### 3. Contracts
|
||||
|
||||
- Resolve one active Compose server/database pair and verify its build, PGDATA,
|
||||
storage mount and database identity before backup. Do not use fixed container
|
||||
names or repository-relative runtime data.
|
||||
- Parse Docker mount identity from `docker inspect` JSON and require exactly one
|
||||
destination match. Do not split human-formatted Go-template output on tabs or
|
||||
spaces: `println` inserts argument spacing that can make a visually correct
|
||||
line fail exact path comparison on a real daemon.
|
||||
- Database connection values used by the automatic wrapper must be exported in
|
||||
an isolated child shell before `capture_command`; do not invoke `env` with a
|
||||
complete `DATABASE_URL` or password assignment in its argv. Docker backup and
|
||||
shadow-restore commands inherit `PGPASSWORD` with
|
||||
`PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD ...`; never use a
|
||||
value-bearing `-e PGPASSWORD=...` Docker argument.
|
||||
- Transfer only a complete `auto-<UTC timestamp>` three-file batch to ARM
|
||||
inbox. The forced key allows write-only `rrsync` or an exact finalize command;
|
||||
it rejects shell, sender/read and destructive rsync options.
|
||||
- ARM revalidates manifest identity, exact regular-file set, sizes and SHA-256,
|
||||
then atomically moves the batch to archive. Archive and weekly directories are
|
||||
outside the rsync root. A different run ID for the same UTC day may replace
|
||||
that day's archive only after full candidate validation: rename the old daily
|
||||
point to a same-filesystem rollback name, publish the candidate, rebuild that
|
||||
ISO week's hard-link checkpoint, then remove the rollback. Any publication or
|
||||
weekly-checkpoint failure restores the prior daily point.
|
||||
- After taking the finalize lock and before reading the incoming partial, ARM
|
||||
reconciles interrupted same-filesystem transactions. A valid
|
||||
`.daily-replacement-<UTC-day>.*` is deleted only when one valid canonical
|
||||
archive for that day exists; when the canonical is absent it is renamed back
|
||||
from its own verified manifest identity. Invalid, linked, duplicate or
|
||||
otherwise ambiguous daily state fails closed and preserves every object.
|
||||
- Weekly `.rollback.*` is authoritative only after full batch validation and is
|
||||
restored when the canonical checkpoint is absent. Weekly `.new.*` is never an
|
||||
authoritative source; it may be removed only after a valid canonical,
|
||||
rollback or unique valid daily source is established. Invalid canonical,
|
||||
rollback, duplicate candidate, week mismatch or non-unique rebuild source
|
||||
fails closed without deleting evidence. A reconciled daily point forces its
|
||||
ISO-week checkpoint to be rebuilt before finalize continues. When a valid
|
||||
canonical daily point and its old rollback coexist, defer deleting that daily
|
||||
rollback until every required weekly reconciliation has succeeded; a weekly
|
||||
validation or rebuild failure must leave the daily rollback in place.
|
||||
- Write production `last-success.json` and prune source copies only after ARM
|
||||
finalize succeeds. Keep seven automatic source/ARM daily points and four ARM
|
||||
weekly checkpoints; unrelated manual batches are outside automatic retention.
|
||||
- Daily, freshness and quarterly reminder timers are separate. Alert summaries
|
||||
contain categories only. A shared Python helper reads the complete WeCom group
|
||||
robot URL only from a regular, executing-account-owned `0600` file and never
|
||||
prints it. It sends the first failure, worsening/new findings, one recovery
|
||||
notice and the first quarterly reminder, while suppressing an identical
|
||||
repeated state. It does not call the OA server container or resolve an employee.
|
||||
- ARM runs its own 26-hour dead-man as root. It verifies the newest archive's
|
||||
exact file set, manifest identity, sizes, checksums and timestamp, so a stopped
|
||||
production host/server can still cause a group alert. Missing, corrupt, stale
|
||||
or future-dated archives fail; recovery sends once.
|
||||
- Production command diagnostics for paired backup, rsync and finalize stay only
|
||||
in the state directory as `last-error.log` mode `0600`. The summary and group
|
||||
message contain only the safe phase category. A later successful off-host run
|
||||
removes the retained diagnostic.
|
||||
- The production freshness check hard-fails missing, malformed, future, stale or
|
||||
Compose-project-mismatched state. A fresh recovery point with a different
|
||||
build SHA is `PASS_WITH_FINDINGS`, because deploy-time code identity and
|
||||
recoverable data freshness are separate facts.
|
||||
- Installer `install` is disabled-by-default; `enable`, real transfer, real
|
||||
WeCom alert and all server/account/key changes remain `EXTERNAL_GATED`.
|
||||
|
||||
### 4. Validation & Error Matrix
|
||||
|
||||
- Source mount/build/project drift, lock conflict or incomplete backup -> fail
|
||||
before success state and source retention.
|
||||
- Transfer/finalize failure, wrong/extra/link artifact or bad hash -> leave no
|
||||
off-host success state, preserve the source batch and retain a root-only local
|
||||
diagnostic. A corrupt same-day candidate leaves the previous archive intact.
|
||||
- Valid same-day retry -> replace only that UTC day's point and rebuild its ISO
|
||||
weekly checkpoint; concurrent finalize -> `flock` rejection.
|
||||
- Interrupted daily replacement with valid canonical -> remove the verified old
|
||||
rollback only after rebuilding the weekly point; canonical missing -> restore
|
||||
the verified rollback; canonical invalid, weekly rebuild failure or candidates
|
||||
ambiguous -> preserve the remaining rollback evidence and fail.
|
||||
- Interrupted weekly replacement with valid canonical -> remove safe staging
|
||||
and verified rollback; canonical missing + valid rollback -> restore it;
|
||||
staging-only + one valid daily source -> rebuild; any invalid or ambiguous
|
||||
authority -> preserve all objects and fail.
|
||||
- Missing, corrupt, stale, future or project-mismatched success record -> hard
|
||||
watchdog failure. Fresh build mismatch -> one finding without a failing exit.
|
||||
- Missing/non-regular/symlink/non-`0600` Webhook file -> fail closed without
|
||||
printing the file value. Identical alert state -> update/evaluate without send.
|
||||
- Ordinary shell, rsync sender/read/delete or archive overwrite -> receiver
|
||||
rejection without changing archive.
|
||||
- ARM host/network/WeCom all unavailable at once -> the group robot path cannot
|
||||
prove its own outage; this physical boundary is not a green state.
|
||||
|
||||
### 5. Good/Base/Bad Cases
|
||||
|
||||
- Good: one verified active source creates a complete batch, write-only rsync
|
||||
resumes inside inbox, ARM finalize verifies and archives it, then production
|
||||
writes success and prunes old automatic source points. If production later
|
||||
stops, ARM independently raises one dead-man alert and one recovery notice.
|
||||
- Base: local implementation and synthetic contracts pass while both server
|
||||
installers remain disabled and explicitly externally gated.
|
||||
- Bad: send through `compose exec server`, hard-code a personal employee number,
|
||||
persist a robot URL in `.env`/unit/repository, give production a normal ARM
|
||||
login key, put archive below inbox, accept a new host fingerprint automatically,
|
||||
or prune local points before target-side finalize.
|
||||
|
||||
### 6. Tests Required
|
||||
|
||||
- Shell syntax plus `run-automatic-backup-contract.sh` using temporary files and
|
||||
mocked Docker/SSH boundaries; it must never contact real hosts.
|
||||
- Docker mocks must return the real `{{json .Mounts}}` shape consumed by the
|
||||
runner. A simplified tab-delimited mock is not valid evidence for real mount
|
||||
parsing.
|
||||
- Contract assertions for same-day replacement, corrupt-candidate preservation,
|
||||
weekly hard-link rebuild, daily/weekly interrupted-transaction recovery,
|
||||
rollback preservation when weekly reconciliation fails, ambiguous-state
|
||||
preservation, source diagnostic mode, build mismatch finding, Webhook
|
||||
mode/symlink rejection and group alert behavior without a server container.
|
||||
Test transport must be explicitly gated and local-only.
|
||||
- Mock Docker assertions must prove synthetic connection values are absent from
|
||||
wrapper/backup/restore argv while an inherited `PGPASSWORD` reaches the
|
||||
mocked container command. Test-fixture `env KEY=value` launch arrays do not
|
||||
need mechanical rewriting because they contain only synthetic values.
|
||||
- Environment contract assertions for both helpers, both installers, production
|
||||
timers, ARM dead-man timer, explicit project/data/build/SSH/Webhook-file
|
||||
binding, disabled-by-default installation and absence of a literal complete
|
||||
robot URL in tracked or untracked repository files.
|
||||
- Real ARM setup, first paired batch, deliberate source-timer stop or transfer
|
||||
failure, dead-man first/repeated/recovery behavior and quarterly shadow restore
|
||||
are separately approved operational evidence. A direct robot send test alone
|
||||
is not dead-man evidence.
|
||||
|
||||
### 7. Wrong vs Correct
|
||||
|
||||
#### Wrong
|
||||
|
||||
```bash
|
||||
compose exec server send-alert --recipient=<personal-employee-number>
|
||||
rsync -a <batch>/ backup-host:<archive>/
|
||||
docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" db pg_dump ...
|
||||
```
|
||||
|
||||
#### Correct
|
||||
|
||||
```bash
|
||||
# Install only with explicit active identity and a root-only Webhook file path;
|
||||
# default remains disabled.
|
||||
<explicit-project-data-build-ssh-and-webhook-file-env> \
|
||||
bash deploy/trialrun/install-automatic-backup-timer.sh install
|
||||
# ARM separately verifies archive freshness even when production is unavailable.
|
||||
python3 scripts/check-offhost-backup-deadman.py <explicit-safe-path-arguments>
|
||||
# The runner writes success and prunes only after restricted target finalize.
|
||||
bash scripts/run-automatic-backup.sh run
|
||||
# The Docker CLI inherits the value without placing it in argv.
|
||||
PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
|
||||
```
|
||||
|
||||
@@ -168,9 +168,10 @@ WeCom leave(category=调休) -> WeCom approval -> OA minimum read-only history
|
||||
- `login|bind` always keeps `snsapi_base`; only explicit `onboard` uses
|
||||
`snsapi_privateinfo`.
|
||||
- The member must be visible to the app, active, return the same userid from
|
||||
both identity calls, and authorize a valid phone plus personal email or
|
||||
business email. A blank/invalid personal email may fall back to a valid
|
||||
business email.
|
||||
both identity calls, and authorize a valid personal email or business email.
|
||||
Phone may be absent; when returned it must be normalized and checked for
|
||||
uniqueness. A blank/invalid personal email may fall back to a valid business
|
||||
email.
|
||||
- `user_ticket`, OAuth code, access token and raw provider responses exist only
|
||||
in request memory. Do not persist them or include them in Cookie, audit,
|
||||
response details or logs; provider/network failures use fixed safe codes.
|
||||
@@ -190,8 +191,9 @@ WeCom leave(category=调休) -> WeCom approval -> OA minimum read-only history
|
||||
- Provider/network failure -> `wecom_api_failed` or safe auth error; no raw
|
||||
payload, token or ticket in output/logs.
|
||||
- Userid mismatch -> `identity_mismatch`; inactive member -> `member_inactive`.
|
||||
- Missing/invalid phone, both emails invalid, or unresolved department ->
|
||||
`missing_required_field`.
|
||||
- Both emails invalid or unresolved department -> `missing_required_field`.
|
||||
- Phone absent -> continue with `NULL`; phone present but conflicting ->
|
||||
`employee_conflict`.
|
||||
- Multiple employees, unique-field race or cross-binding -> `employee_conflict`.
|
||||
- Department binding points to missing/archived data -> `department_invalid`;
|
||||
multiple same-name OA departments -> `department_ambiguous`.
|
||||
@@ -232,3 +234,92 @@ ordinary login -> snsapi_base
|
||||
one-time onboard -> explicit private consent -> safe unique employee-only write
|
||||
business role assignment -> separate administrator-audited operation
|
||||
```
|
||||
|
||||
## Scenario: WeCom Directory Export With Optional Phone
|
||||
|
||||
### 1. Scope / Trigger
|
||||
|
||||
- Trigger: importing an administrator-exported WeCom directory XLSX, or
|
||||
changing `Employee.phone` across database, API, HR sync, OAuth and UI layers.
|
||||
- Purpose: allow a real member to remain temporarily without a phone while
|
||||
preserving stable identity, uniqueness and privacy controls.
|
||||
|
||||
### 2. Signatures
|
||||
|
||||
- DB: `Employee.phone String? @unique`; migration drops only the `NOT NULL`
|
||||
constraint and keeps the unique index.
|
||||
- CLI: `npm run import:wecom-directory-export -- --input=- --dry-run`.
|
||||
- Apply: `WECOM_DIRECTORY_EXPORT_APPLY=1 ... --input=- --apply
|
||||
--actor-employee-no=<employee-no>`.
|
||||
- User API and frontend employee/user types expose `phone: string | null`;
|
||||
create/update accepts an omitted or explicit `null` phone.
|
||||
|
||||
### 3. Contracts
|
||||
|
||||
- Missing phone is stored as SQL `NULL`, never `""`, a repeated placeholder or
|
||||
a synthesized fake number. Multiple `NULL` rows are valid; non-null phones
|
||||
remain unique.
|
||||
- WeCom account and valid enterprise email remain required for directory-export
|
||||
onboarding. Account, non-null phone and enterprise email duplicates block
|
||||
the input.
|
||||
- An empty source phone never clears an existing OA phone. Only an explicitly
|
||||
supplied valid phone participates in update and unique-conflict checks.
|
||||
- The XLSX is accepted only from stdin and defaults to dry-run. Apply requires
|
||||
both `--apply` and `WECOM_DIRECTORY_EXPORT_APPLY=1`, plus an active actor.
|
||||
- CLI output contains counts and categories only. The workbook, full local path,
|
||||
names, accounts, phones, emails, departments and raw rows must not enter Git,
|
||||
Trellis artifacts or reports.
|
||||
- New accounts receive only `employee`; finance, CEO and admin roles remain a
|
||||
separate administrator-audited operation.
|
||||
|
||||
### 4. Validation & Error Matrix
|
||||
|
||||
- Missing phone cell -> valid profile with `phone=null`.
|
||||
- Supplied malformed phone -> incomplete/blocked; do not downgrade it to an
|
||||
absent phone and do not write it.
|
||||
- Missing/invalid enterprise email, account, name or department -> skipped or
|
||||
blocked as `missing_required_field`.
|
||||
- Duplicate account/non-null phone/enterprise email in the workbook ->
|
||||
`duplicate_values`; error output contains counts only.
|
||||
- Existing OA unique-field or binding conflict -> conflict; no automatic merge.
|
||||
- `--apply` without confirmation env or active actor -> refuse before sync.
|
||||
- Migration rollback with any `phone IS NULL` row -> stop; do not fill fake
|
||||
values merely to restore `NOT NULL`.
|
||||
|
||||
### 5. Good/Base/Bad Cases
|
||||
|
||||
- Good: five members have unique phones, one member has no phone, all six have
|
||||
unique accounts and enterprise emails; dry-run plans six records and reports
|
||||
only aggregate counts.
|
||||
- Base: a later export still omits a member phone; the member's existing OA
|
||||
phone is preserved.
|
||||
- Bad: write an empty string or shared placeholder, use personal email in place
|
||||
of the required enterprise email for bulk identity, or auto-grant a business
|
||||
role during import.
|
||||
|
||||
### 6. Tests Required
|
||||
|
||||
- Prisma validation plus a migration review proving only `DROP NOT NULL`.
|
||||
- Unit tests for instruction rows/Chinese headers, five-present/one-missing,
|
||||
malformed phone, missing enterprise email and duplicate account/phone/email.
|
||||
- Route/service tests for create/update/import `null`, non-empty unique checks,
|
||||
OAuth missing phone and empty-source preservation of an existing phone.
|
||||
- Frontend typecheck/build and displays that render a clear empty-phone label.
|
||||
- Full backend regression, Trellis governance, sensitive-content scan and
|
||||
`git diff --check`; a real workbook may be parsed only into aggregate counts
|
||||
and remains Git-ignored.
|
||||
|
||||
### 7. Wrong vs Correct
|
||||
|
||||
#### Wrong
|
||||
|
||||
```js
|
||||
phone: source.phone || "00000000000"
|
||||
```
|
||||
|
||||
#### Correct
|
||||
|
||||
```js
|
||||
phone: source.phone || null
|
||||
// Existing employee updates omit `phone` entirely when the source is empty.
|
||||
```
|
||||
|
||||
@@ -108,6 +108,15 @@ GET /auth/wecom/login?mode=onboard
|
||||
- 已有绑定员工只补齐经授权的规范化资料且不改角色；新员工在单事务内创建 employee、`employee` 角色、企微绑定和脱敏审计。任何一步失败均不留下部分账号。
|
||||
- 自助开通不替代管理员职责分配；`finance`、`ceo` 仍沿用角色接口的 token version、会话撤销和审计合同。
|
||||
|
||||
## 企微通讯录导出与手机号可空
|
||||
|
||||
- 管理员导出的 XLSX 只作为一次性受控输入，不进入 Git、Trellis、报告或生产文件系统；CLI 支持从 stdin 读取，默认仅 dry-run。
|
||||
- 入口识别企微官方说明行和中文表头，将姓名、账号、职务、部门、手机、邮箱/企业邮箱、激活/禁用状态归一化为既有 HR master profile，再复用部门解析、员工匹配、企微绑定、审计和 `employee` 初始角色语义。
|
||||
- `Employee.phone` 改为 nullable unique：缺失值统一保存为 `NULL`，非空值继续唯一。企业邮箱仍必填且唯一，企微账号仍是绑定主键；不生成占位手机号。
|
||||
- 来源未提供手机号时，不得把既有 OA 手机号清空；来源提供手机号时才进入变更和唯一性检查。本人授权同样允许手机号缺失，但邮箱或企业邮箱仍必须有效。
|
||||
- apply 需要命令参数和独立环境变量双门，且真实运行另行批准；dry-run 与 apply 的报告都只含计数、动作、冲突类别和脱敏摘要。
|
||||
- 生产迁移回滚前必须确认不存在 `phone IS NULL`；否则停止回滚，不用假数据填充。
|
||||
|
||||
## 部署目录统一设计约束
|
||||
|
||||
- 现场是混合部署，不能把“统一目录”处理成普通 rsync：必须以当前活动 PostgreSQL 数据和附件挂载为唯一数据事实源。
|
||||
@@ -131,4 +140,4 @@ GET /auth/wecom/login?mode=onboard
|
||||
2. 本地先修复 Compose 可配置回环端口及部署脚本动态端口探测，通过全部门禁并提交、发布批准提交。
|
||||
3. 停机后把活动 PGDATA 和附件复制到新的显式绝对目录，复制前目标必须为空；保留旧目录原位且不再挂载写入，直到观察期结束。复制仅在旧 PostgreSQL 完全停止后执行，并保留所有权、权限、链接和扩展属性。
|
||||
4. 用新 Compose 重建 db/init/server，应用已提交 migrations；旧容器和旧目录在验收前保留，不清理。
|
||||
5. 公网构建标识、数据计数、角色、企微入口和关键只读检查通过后更新运维 timer；失败则停止新容器并恢复旧目录/容器和原 Caddy 目标。
|
||||
5. 公网构建标识、数据计数和关键部署只读检查通过后更新运维 timer；角色与企微入口仍按各自真实写入门单独验收，不阻塞部署运维基线收口。部署检查失败则停止新容器并恢复旧目录/容器和原 Caddy 目标。
|
||||
|
||||
@@ -20,15 +20,18 @@
|
||||
- [ ] 复用既有 HR 同步与角色语义写入 `employee + finance`、`employee + ceo`，复核会话失效、审计和 maker-checker，不部署、不迁移。
|
||||
- [x] 增加默认关闭的一次性企微敏感资料授权入口，保持普通登录静默；实现 `user_ticket -> getuserdetail`、active/字段/唯一性门禁和仅 `employee` 自助开通。
|
||||
- [x] 补齐 OAuth 路由、服务、配置契约和单元测试，证明票据/原始敏感响应不进入日志、Cookie、数据库或审计。
|
||||
- [ ] 修复部署脚本的现场端口解析，确定唯一部署目录，并在活动 DB/附件挂载、环境摘要和旧镜像回滚点全部确认后才执行获批切换。
|
||||
- [x] 修复部署脚本的现场端口解析，确定唯一部署目录，并在活动 DB/附件挂载、环境摘要和旧镜像回滚点全部确认后执行获批切换。
|
||||
- [x] 完成本地 Compose/备份加固：移除固定容器名和 DB 宿主端口，参数化数据目录，deploy/timer 显式 project，备份来源缺失硬失败，远端构建标识改由容器内 Node 核验。
|
||||
- [x] 增加隐式附件源、隐式 DB 容器、DB 端口与固定容器名回退的负向契约演练；未部署或启用新 Compose。
|
||||
- [x] 收紧正式 deploy 数据路径：显式绝对路径、`existing|initialize` 模式、既有 PGDATA 标记和禁止 Docker 静默建空目录。
|
||||
- [x] 让成对 manifest 尊重显式 `BUILD_SHA` 并支持活动镜像 digest；补 Node/Python 等价与非法来源标识负向测试。
|
||||
- [x] 把旧 Compose project 实测、镜像 digest、58 项迁移与关键表计数、影子逐表/附件计数验收、停机后复制并保留旧目录写入 runbook/checklist/spec。
|
||||
- [ ] 部署后验证唯一 Compose 来源、构建标识、迁移状态、Caddy 公网 `/ready` 和前端 marker；不因服务可访问而忽略来源漂移。
|
||||
- [x] 部署后验证唯一 Compose 来源、构建标识、迁移状态、Caddy 公网 `/ready` 和前端 marker；不因服务可访问而忽略来源漂移。
|
||||
- [ ] 用户在企微应用详情选择手机、邮箱、企业邮箱敏感字段；部署开关开启后，两名目标分别通过一次性链接本人确认授权。
|
||||
- [ ] 两人自助开通成功后写入获批 `finance` / `ceo` 职责，复核不同身份、maker-checker、会话撤销、审计和付款零写入。
|
||||
- [x] 将 `Employee.phone` 改为 nullable unique，并同步员工创建/批量导入、企微 HR master、本人授权、登录查询和前端空值合同；不执行生产 migration。
|
||||
- [x] 增加企微通讯录 XLSX/stdin 默认 dry-run 入口，复用 HR master 写入语义，apply 使用独立双门且本阶段不执行。
|
||||
- [x] 用完全合成的通讯录覆盖“5 个手机号 + 1 个缺失”、重复账号/手机号/邮箱、说明行/中文表头、既有手机号不被空来源清除和报告隐私门禁。
|
||||
|
||||
## 推荐文件边界
|
||||
|
||||
@@ -52,6 +55,7 @@
|
||||
- `cd server && npm run prisma:validate`
|
||||
- `cd server && npm test -- --run tests/unit/routes/wecomAuth.test.js tests/unit/wecomHrMaster.service.test.js`
|
||||
- `cd server && npm test -- --run tests/unit/routes/wecomAuth.test.js tests/unit/wecomHrMaster.service.test.js tests/unit/wecomOAuthOnboarding.service.test.js`
|
||||
- `cd server && npm test -- --run tests/unit/routes/users.import.test.js tests/unit/wecomHrMaster.service.test.js tests/unit/wecomOAuthOnboarding.service.test.js tests/unit/scripts/import-wecom-directory-export.test.js`
|
||||
- `bash -n deploy/trialrun/deploy.sh`
|
||||
- `bash -n deploy/trialrun/install-warch3-ops-timer.sh`
|
||||
|
||||
@@ -71,6 +75,22 @@
|
||||
- 已从活动 DB 容器与活动 server 附件挂载生成正确批次并复制到 ARM 备份机；DB catalog、manifest、源/目标产物 SHA-256、附件 tar 与 0 个普通文件计数均通过，且未伪造旧实例 build SHA。ARM 只承担异机备份与恢复演练职责，不运行 OA 服务。
|
||||
- 已在 ARM 备份机用不发布端口的一次性 PostgreSQL 16 容器恢复正确批次：58 项成功迁移、12 张关键表计数和 0 个附件普通文件均与冻结基线一致，包含镜像拉取的实测 RTO 为 33 秒。演练后容器、临时附件目录和本轮新增镜像均已删除，未运行 OA server；应用 `/ready` 留待生产部署环境验证。
|
||||
|
||||
## 手机号可空与通讯录导入本地证据（2026-07-27）
|
||||
|
||||
- 后端全量 Vitest：163 个测试文件、1016 个测试通过；补充双 apply 门断言后，通讯录导入聚焦 8 项再次通过。新增覆盖普通用户创建/显式清空、CSV 空手机号、HR 空来源保留既有值、OAuth 无手机号建档、通讯录说明行/中文表头、企业邮箱门禁、非法手机号、重复字段、apply 双门及脱敏状态。
|
||||
- `prisma validate`、四个后端改动文件语法、前端 typecheck/build、Trellis task/governance、敏感扫描、流程合同、文档注册审计和 `git diff --check` 通过。
|
||||
- 管理员提供的真实通讯录仅执行解析级只读检查：6 名 active、5 个手机号、1 个缺失、6 个企业邮箱、0 个不完整；未连接数据库、未执行 apply、未分配角色、未发送企微消息，文件保持 Git 排除。
|
||||
- 最终只读复核确认 migration 仅删除手机号 `NOT NULL`、唯一索引保留；导入及本人开通只创建 `employee`，不自动授予 `finance`、`ceo`、`admin` 或 `system_admin`；空来源更新省略 phone 字段，真实文件名、完整路径和个人字段未进入差异。
|
||||
|
||||
## 生产部署与 timer 收口证据（2026-07-27）
|
||||
|
||||
- 已在获批停机窗口把生产切换到唯一 Compose project；活动 DB 与附件从停机冻结副本进入新的独立目录，旧容器保持停止，旧数据目录原位保留作回滚点。
|
||||
- 生产已应用 61 项 migration；12 张关键表计数与冻结基线一致，活动附件普通文件数仍为 0。server 仅回环发布，DB 不发布宿主端口。
|
||||
- 容器内与公网 `/ready`、前端 marker 均返回批准提交的同一 12 位构建标识；runtime smoke、external-deps smoke 和公网 bundle/marker 检查通过。
|
||||
- 运维 timer 安装器已要求显式绑定 Compose project、活动 PGDATA、活动附件目录、完整后端 SHA 和 12 位前端 SHA。生产 timer 已重新安装并保持 enabled/active，daily service 未被手工触发；unit 环境与活动挂载/构建标识一致，同环境 Compose config 与 server exec 解析通过。
|
||||
- timer 首次收口时发现从 deploy 复制的旧路径正则会误拒绝所有绝对路径；安装在写 unit 前安全停止。现已与 deploy 的根目录、尾随斜杠、双斜杠及点路径段门禁保持一致，并由环境契约阻止旧正则回退。
|
||||
- 本轮没有启动真实告警任务，没有修改员工、角色、付款、企微后台或企微开通开关；职责分离与两人本人授权继续保持未完成。
|
||||
|
||||
## 回滚点
|
||||
|
||||
- 成对入口失败：保留四个底层脚本，不写完整 manifest。
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
- 经用户逐项授权后，修正企微通讯录与人事助手的双 Secret 读取分工，并用脱敏 dry-run 唯一确认两名既有企微成员。
|
||||
- 唯一匹配且必填资料、冲突检查通过后，将两名成员同步为 OA 员工，并分别配置 `employee + finance` 与 `employee + ceo`，形成不同自然人的付款制单/复核职责。
|
||||
- 增加独立的一次性企微敏感资料授权入口：普通登录继续使用 `snsapi_base`，首次开通使用 `snsapi_privateinfo`；未绑定的在职可见成员授权成功后只自动获得 `employee` 角色。
|
||||
- 支持管理员提供的企微通讯录 XLSX 通过 stdin 做脱敏 dry-run，并允许个别成员暂缺手机号；企业邮箱、企微账号、姓名和部门仍作为建档门禁。
|
||||
- 修复部署脚本与现场端口、目录、容器和构建标识契约，统一真实运行来源；切换前保留活动数据、环境文件和旧容器回滚点。
|
||||
|
||||
## Requirements
|
||||
@@ -37,10 +38,11 @@
|
||||
- 将真实内部实例核查列为后续同任务外部门禁：运行 commit/访问方式/TLS、真实职责分配、异机副本和恢复后 `/ready` 均需另行批准和脱敏留证。
|
||||
- 企微通讯录 Secret 只用于 `/user/list_id`；自建应用 Secret 用于 `/user/get`、`/department/list` 和获授权的人事助手 `/hr/get_staff_info`，不得回退到已废弃的人事助手 Secret。
|
||||
- 真实同步前只输出匹配数量、身份是否不同、缺失字段数、冲突数和计划创建/更新数，不输出姓名、userid、手机号、邮箱、部门原文或原始 payload。
|
||||
- `Employee.phone` 允许为 `NULL`，但不得写空字符串或假手机号；非空手机号继续保持唯一。企微账号和企业邮箱必须非空且唯一，缺手机号成员使用企微或邮箱登录，后续可由本人授权或管理员补齐。
|
||||
- 真实写入必须复用现有员工同步、角色、会话撤销和审计语义；财务处理人与 CEO 复核人必须是两个不同企微身份，且不得把 `admin` 当作普通业务复核角色。
|
||||
- 敏感资料授权必须显式受 `WECOM_OAUTH_ONBOARDING_ENABLED` 控制，默认关闭；成员必须在应用可见范围、企微状态 active，并实际同意手机号及邮箱或企业邮箱授权。
|
||||
- 敏感资料授权必须显式受 `WECOM_OAUTH_ONBOARDING_ENABLED` 控制，默认关闭；成员必须在应用可见范围、企微状态 active，并至少实际同意邮箱或企业邮箱授权；手机号可暂缺，但返回时必须规范化并参与唯一性检查。
|
||||
- `user_ticket` 只在服务端内存中用于一次 `getuserdetail` 调用，不写日志、Cookie、数据库、审计或错误详情；授权 code 和原始敏感响应同样不得持久化。
|
||||
- 自助开通按企微 userid、手机号和邮箱做唯一性检查；新账号只创建 `employee` 角色，随机不可知密码并要求改密，不自动授予财务、CEO、管理员或审批角色。
|
||||
- 自助开通按企微 userid、非空手机号和邮箱做唯一性检查；新账号只创建 `employee` 角色，随机不可知密码并要求改密，不自动授予财务、CEO、管理员或审批角色。
|
||||
- 普通 `snsapi_base` 登录、已绑定账号登录和现有绑定流程保持兼容，不因资料授权功能改成每次手工确认。
|
||||
- 生产统一以仓库既定部署目录和 Compose 契约为目标，但活动 PostgreSQL 与附件数据是唯一数据事实源；停机切换不得复制运行中的 PostgreSQL 文件、启动第二套空库或覆盖环境变量。
|
||||
- Compose 项目名、容器身份与 DB/附件目录必须在同一次获批停机窗口切换；DB 不发布宿主端口，备份不得猜测容器名或附件目录。
|
||||
@@ -62,18 +64,19 @@
|
||||
- [ ] 获批写入后，两名活跃员工分别具有 `employee + finance` 与 `employee + ceo`，企微绑定不同，角色审计存在且未产生付款业务写入。
|
||||
- [x] 普通企微登录仍生成 `snsapi_base`，一次性开通入口生成 `snsapi_privateinfo`；缺少开关、`user_ticket`、成员授权字段、active 状态或唯一性时均拒绝创建（本地 synthetic 验证）。
|
||||
- [ ] 一次性授权通过 `getuserdetail` 获取成员同意的手机和邮箱，自动创建或补齐 OA 员工与企微绑定；新建账号角色严格等于 `employee`，日志和审计不含票据或完整敏感响应。
|
||||
- [ ] 部署目录、Compose、环境文件、运维 timer 和活动容器来源统一；活动 DB/附件连续，远端 migrations 与批准提交一致，公网 `/ready` 和前端 marker 返回同一短构建标识。
|
||||
- [x] 部署目录、Compose、环境文件、运维 timer 和活动容器来源统一；活动 DB/附件连续，远端 migrations 与批准提交一致，公网 `/ready` 和前端 marker 返回同一短构建标识。
|
||||
- [x] 本地 Compose、部署、timer 和备份合同完成加固并覆盖负向回退；未把本地通过误报为生产切换完成。
|
||||
- [x] 部署入口拒绝隐式/相对数据目录及缺少数据模式，成对 manifest 可记录显式 build SHA 或活动镜像 digest，并有 Node/Python 等价测试。
|
||||
- [ ] 两名目标成员分别完成一次本人授权后，账号唯一、企微绑定不同，再由管理员写入 `employee + finance` 与 `employee + ceo`；maker-checker、会话撤销和角色审计通过。
|
||||
- [x] 本地合成企微导出证明 5 人有手机号、1 人手机号为空时 dry-run 可通过；空值保存为 `NULL`，非空手机号/邮箱/企微账号重复均被拒绝，报告不含个人字段原文。
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 不读取真实附件正文，不执行生产恢复、数据库 reset、迁移历史改写或付款/报销业务写入。
|
||||
- 不修改 Prisma schema、公开业务 API、权限模型、审批链或付款状态机；仅修复既有 HR 同步服务的凭证分工，并通过既有角色模型写入获批职责。
|
||||
- 除已批准的 `Employee.phone` nullable 兼容外，不扩大修改其他 Prisma schema、权限模型、审批链或付款状态机；公开用户 API 仅同步手机号可空合同，并通过既有角色模型写入获批职责。
|
||||
- 不引入 MinIO、云备份 SDK、全量灾备平台、自动跨地域复制或 24×7 值守。
|
||||
- 不在仓库提交备份文件、恢复数据、环境变量、凭证、真实账号或完整本地路径。
|
||||
- 不发送企微消息、不创建企微审批模板、不把手机号/邮箱改为可空、不写占位假数据，也不将敏感授权变成普通登录的强制步骤。
|
||||
- 不发送企微消息、不创建企微审批模板、不把邮箱改为可空、不写占位假数据，也不将敏感授权变成普通登录的强制步骤。
|
||||
- 不在本任务完成企微调休、导航、报销或模块退役；它们属于后续批次。
|
||||
|
||||
## Test Plan
|
||||
@@ -97,13 +100,16 @@
|
||||
|
||||
先改进可恢复性和防误操作，再接触真实环境。默认工作流只支持影子恢复；真实恢复不是普通脚本路径，而是需要停机、负责人确认、成对备份、回滚点和独立授权的事故操作。
|
||||
|
||||
企微敏感字段采用“日常静默登录 + 一次性本人授权开通”的双入口。批量同步不猜测敏感资料，不修改全局员工字段可空性；自助开通只授予 `employee`，业务角色继续由管理员审计写入。
|
||||
企微敏感字段采用“日常静默登录 + 一次性本人授权开通”的双入口。批量同步不猜测敏感资料；手机号按已批准范围改为全局 nullable，空值统一保存为 `NULL`。自助开通只授予 `employee`，业务角色继续由管理员审计写入。
|
||||
|
||||
## Local Evidence
|
||||
|
||||
- 已使用一次性 PostgreSQL 16 容器和合成附件完成多轮往返演练，最近一次本地参考 RTO 为 6 秒。
|
||||
- 演练覆盖 manifest/校验和、错误确认、正式风格库名、损坏 DB、损坏 tar、非空附件目录、默认试运行附件目录和危险 tar 路径拒绝。
|
||||
- 该证据不包含真实业务数据、真实运行实例、异机副本或部署后 `/ready`，因此最后一项验收保持未完成。
|
||||
- 2026-07-27 完成手机号 nullable 本地切片：Prisma 仅新增 `phone DROP NOT NULL` migration，用户创建/更新/批量导入、HR master、本人授权和前端空值合同已同步；本轮没有执行 migration 或写入真实员工。
|
||||
- 合成通讯录覆盖说明行/中文表头、5 个手机号加 1 个缺失、非法手机号、缺企业邮箱、重复账号/非空手机号/企业邮箱、双重 apply 门和脱敏报告。管理员提供的真实 XLSX 仅在内存中只读解析为 6 名 active、5 个手机号、1 个缺失、6 个企业邮箱、0 个不完整；未连接数据库，文件保持 Git 排除。
|
||||
- 最终本地门禁：后端全量 163 个测试文件、1016 个测试通过；补充双 apply 门测试后，通讯录导入聚焦 8 项再次通过。Prisma validate、前端 typecheck/build、Trellis task/governance、敏感扫描、流程合同、文档注册审计和 `git diff --check` 通过。文档审计保留仓库既有 22 个未注册文件 warning，无新增 error。
|
||||
|
||||
## Live Read-Only Evidence
|
||||
|
||||
|
||||
@@ -111,3 +111,12 @@
|
||||
- 已使用正确异机批次在不发布端口的一次性 PostgreSQL 16 容器中完成真实备份格式影子恢复；58 项成功迁移、12 张关键表计数与冻结基线逐项一致，附件归档安全检查和解包后的 0 个普通文件也一致，包含镜像拉取的实测 RTO 为 33 秒。
|
||||
- 恢复演练后，一次性数据库容器、临时附件目录和本轮新增 PostgreSQL 镜像均已删除；ARM 继续只承担备份与恢复演练职责，没有运行 OA server 或开放 OA 端口。
|
||||
- 当前证据证明 DB＋附件异机恢复点可用，但不替代部署环境的应用 `/ready`、构建标识、Caddy 和前端 marker 验收。生产服务未停止，数据库、附件、容器、Caddy、timer、角色和企微均未修改。
|
||||
|
||||
## 2026-07-27 获批生产切换与运维收口
|
||||
|
||||
- 已完成获批生产切换：新 Compose project 成为唯一活动来源，旧项目容器保持停止；旧数据目录和旧容器均保留，正式活动数据使用停机后复制的新目录。
|
||||
- 生产共 61 项 migration 成功，12 张关键表计数与冻结基线一致，活动附件普通文件数为 0；server 只绑定回环地址，DB 未发布宿主端口。
|
||||
- 容器内与公网 `/ready`、前端 marker 均观察到批准提交的同一 12 位构建标识；runtime、外部依赖和公网资源标识检查通过。
|
||||
- 运维 timer 已重新安装并显式绑定当前 Compose project、活动 DB/附件目录和前后端构建标识；timer 为 enabled/active，daily service 保持 inactive，未手工触发真实告警。使用同一组脱敏环境事实执行 Compose config 与 server exec 均通过。
|
||||
- timer 安装前的路径与构建门禁先在本地通过；一次旧正则误拒绝绝对路径后，安装在 unit 写入前停止，随后修复并增加防回退契约。全程未输出或写入 Secret、完整宿主路径和环境原文。
|
||||
- 部署与恢复基线现已完成；真实职责分配、两名成员本人敏感资料授权和企微开通仍是独立未完成门禁，因此整体仍不得宣称付款 maker-checker 或企微自助开通已可用。
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,85 @@
|
||||
# 每日自动异机备份设计
|
||||
|
||||
## 总体边界
|
||||
|
||||
本任务不建设备份平台。生产主机负责生成和推送一个不可变成对批次，ARM 用受限 SSH 接收身份把传输区与归档区隔离；systemd 负责每日执行、缺失监测和季度提醒。
|
||||
|
||||
```text
|
||||
production 02:30
|
||||
resolve active Compose identity
|
||||
-> flock
|
||||
-> backup-all.sh (DB + storage + manifest)
|
||||
-> local verify
|
||||
-> rsync to ARM inbox/<run-id>.partial
|
||||
-> restricted finalize <run-id> <manifest-sha>
|
||||
-> ARM verify exact files/manifest/hash
|
||||
-> atomic move to archive/<run-id>
|
||||
-> daily + weekly retention
|
||||
-> write safe success state
|
||||
-> prune local daily copies
|
||||
|
||||
production 07:10 watchdog
|
||||
-> recovery point missing/corrupt/stale/project drift ? fail + group alert
|
||||
-> build SHA differs ? one finding + group alert
|
||||
|
||||
ARM independent watchdog
|
||||
-> newest finalized archive <= 26h ? pass : direct group alert
|
||||
-> source host/server/container not required
|
||||
|
||||
quarterly reminder
|
||||
-> one redacted WeCom reminder per calendar quarter
|
||||
-> human runs ARM shadow restore separately
|
||||
```
|
||||
|
||||
## 生产备份编排
|
||||
|
||||
- 新增一个宿主 wrapper，所有正式输入由安装器写入 systemd：仓库根、Compose project/file/env、活动 PGDATA/附件目录、完整/短构建 SHA、备份根、状态根、SSH alias/key/known-hosts 和告警开关。
|
||||
- wrapper 通过 Compose label 解析唯一 server/db 容器，并把实际 DB/附件 mount、server build SHA 与安装时冻结值逐项比较。任一漂移在调用 `backup-all.sh` 前停止。
|
||||
- DB 名称/用户和密码只从活动容器环境在内存中取得，传给现有 Docker 备份路径；不得 source 漂移的旧部署 env，不得输出取值。
|
||||
- 每个自动批次使用 `auto-<UTC timestamp>`，源端根和状态根均为显式安全绝对路径、权限 `0700`。`flock` 拒绝同一任务重入。
|
||||
- 只有 ARM finalize 成功后才写 `last-success.json` 并清理源端旧日批次。失败保留本次完整源批次供重试，不产生异机成功状态。
|
||||
|
||||
## ARM 受限接收边界
|
||||
|
||||
- ARM 一次性安装 rsync，创建密码锁定的专用账号、独立 inbox/archive/weekly 目录和一把专用公钥。该公钥的 `authorized_keys` 强制执行仓库提供的 receiver，并禁用 PTY、端口转发、agent 转发和 X11 转发。
|
||||
- receiver 只接受两类 `SSH_ORIGINAL_COMMAND`：rsync server 接收模式，或 `hyb-backup-finalize <run-id> <manifest-sha256>`。其他命令全部拒绝。
|
||||
- rsync 通过 `rrsync -wo` 只能进入 inbox；archive 和 weekly 位于 rrsync 根之外。即使生产端密钥泄露，也不能用 rsync 读取或删除已归档批次。
|
||||
- finalize 只接受 `auto-` 安全批次号和完整 manifest SHA。它要求 inbox 中恰有三个普通文件、无链接，manifest 的 runId/status/schema、相对文件名、大小和 SHA-256 全部匹配；同 run_id 重复 finalize 继续保持幂等。
|
||||
- 同 UTC 日已有不同 run_id 时，先完整校验并冻结新批次权限，再把旧日归档 rename 到同文件系统临时回滚名、新批次 rename 到正式归档名；新批次发布失败立即把旧归档 rename 回原名。新批次发布成功后才删除旧回滚名，并重建该 ISO 周硬链接检查点，使每日一个恢复点和周检查点都指向当前有效批次。
|
||||
- archive 保留最近 7 个日批次；每个 ISO 周首次成功批次用只读硬链接快照形成周检查点，保留最近 4 周。清理不得删除最后一个 archive 或当前 finalize 批次。
|
||||
- 旧手工/审计批次不进入自动前缀和自动轮转范围。
|
||||
|
||||
## 传输与主机身份
|
||||
|
||||
- 生产 root 使用一把仅用于本接收器的私钥，权限 `0600`；SSH config 使用专用 alias，known_hosts 由人工核对的固定主机指纹生成，timer 固定 `StrictHostKeyChecking=yes`，不得使用 `accept-new`。
|
||||
- rsync 只发送完整源批次到同名 `.partial` inbox。网络中断可续传同一 partial；远端 archive 同名存在时 finalize 失败，不覆盖。
|
||||
- 仓库、任务和报告只记录 alias 类别、布尔值和批次数量，不记录地址、账号、密钥、公钥原文或完整运行目录。
|
||||
|
||||
## 告警与静默失败检测
|
||||
|
||||
- 新增一个小型宿主告警 helper，使用 Python 标准库从 root-only Webhook 文件读取完整 URL，并向企微运维群机器人发送固定白名单文本；不复用 OA server 容器、员工表或个人工号。
|
||||
- 生产 wrapper 无论成功或失败都生成字段白名单 summary；即时失败和源端 watchdog 直接调用宿主 helper。Webhook 文件缺失、权限不是 `0600`、不是普通文件或内容格式不合法时 fail closed，并在本机诊断中留类别，不打印 URL。
|
||||
- ARM receiver 安装器同时安装独立 dead-man service/timer。watchdog 读取 archive 中最新有效自动批次的完成时间，超过 26 小时、没有批次或 manifest 无法验证时直接调用同一 helper；状态文件实现首次失败、重复降噪和恢复通知。
|
||||
- 生产 stale watchdog 仍验证 schema/status/project/时间。build SHA 不同输出 `PASS_WITH_FINDINGS`，不返回失败；恢复点缺失、损坏、未来时间、超过阈值或 project 不同继续返回失败。
|
||||
- 生产外部命令的 stdout/stderr 只写状态目录下的临时诊断，失败时原子保留为 `last-error.log` 并设为 `0600`，成功后清理本次临时诊断。summary 和企微文本不得复制诊断正文。
|
||||
- Webhook secret 同时存在于两台主机的 root-only 文件，仓库只保存“文件路径”合同；`.env.example` 和 systemd `Environment=` 都不得出现 URL/key。
|
||||
- 季度提醒以年份+季度作为 finding key，确保每季度一次且同季度不重复；只提醒人工影子恢复，不执行恢复。
|
||||
|
||||
## 安装与回滚
|
||||
|
||||
1. 本地先实现合成测试、受限 receiver、告警适配和两个安装器，不接触生产/ARM。
|
||||
2. 获批后先由用户在企微创建运维群机器人，并分别把同一 Webhook 写入两台主机 root-only 文件；不向 Codex、仓库或普通 `.env` 提供其内容。
|
||||
3. 在 ARM 安装 rsync、受限账号/receiver/公钥、dead-man timer 和目标目录；使用无业务数据的合成批次验证拒绝 shell、同日安全取代和 dead-man 状态逻辑。
|
||||
4. 再在生产安装专用私钥/known_hosts、每日备份 service/timer、stale watchdog 和季度提醒；首次安装先保持 timer disabled。
|
||||
5. 手工启动一次获批自动备份，核对源/ARM 哈希、状态和无敏感输出；再 enable timers。
|
||||
6. 真正停止一次源端 timer 或故意让一次传输失败，等待/模拟超过 26 小时并确认 ARM 群告警；恢复链路后验证恢复通知和新批次成功。
|
||||
|
||||
回滚只 disable/stop 新 timer、移除 unit 和专用密钥授权；保留已生成的完整备份、状态报告和 ARM archive，不删除历史数据。旧手工 `backup-all.sh` 入口保持兼容。
|
||||
|
||||
## 取舍
|
||||
|
||||
- 选择生产推送而非 ARM 拉取，是为了让一次任务同时掌握生成、传输、目标校验和即时告警结果。
|
||||
- 不给生产普通 ARM shell，而用 inbox + 强制 receiver + archive 隔离，抵消推送方案最主要的跨机删除风险。
|
||||
- 使用 rsync 而不是新备份服务；当前压缩批次规模较小，收益主要是断点续传和成熟 SSH 传输，不追求块级去重。
|
||||
- 不自动恢复。季度人工影子恢复才是可恢复性证据，日 timer 只负责生成恢复点。
|
||||
- 运维加固在 dead-man、真实失败链路、timer 和现有 runbook 恢复卡完成后硬收口；公司只有一名系统管理员，季度影子恢复由该管理员按卡自查，不设置第二人走查门禁。后续不继续新增平台化备份或告警能力，回到产品 backlog。
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,85 @@
|
||||
# 每日自动异机备份实施计划
|
||||
|
||||
## 实施步骤
|
||||
|
||||
- [x] 实现宿主自动备份 wrapper：安全绝对路径、唯一 Compose 来源、活动 mount/build 校验、`flock`、成对备份、ARM 推送/finalize、成功状态和源端 7 日保留。
|
||||
- [x] 实现 ARM 受限 receiver：只允许 rrsync 写 inbox 和 finalize，严格校验三文件 manifest/大小/哈希、原子归档、7 日 + 4 周保留和最后成功保护。
|
||||
- [x] 实现生产与 ARM 安装器：专用账号/密钥/known_hosts 边界、systemd daily/watchdog/quarterly units、安装/卸载幂等和默认不触发真实任务。
|
||||
- [x] 泛化现有企微运维告警的 stdin summary、issueId 和备份建议，使用独立状态文件并补单元测试。
|
||||
- [x] 增加合成契约测试：来源漂移、锁冲突、不完整/损坏/重复批次、续传 partial、拒绝普通 shell、保留轮转、stale 和季度降噪。
|
||||
- [x] 更新部署环境契约、runbook、checklist、管理员手册的最小操作入口和 `.trellis/spec/hyb-oa/deployment-trialrun.md`；不新增平行长文档。
|
||||
- [x] 生成一页脱敏恢复卡内容并放在现有 runbook 的首部紧急段；由唯一系统管理员维护并用于季度自查，不另建知识库副本或第二人走查门禁。
|
||||
- [x] 完成本地全量门禁并提交；生产、ARM 安装另行请求批准。
|
||||
- [x] 获批后先安装 ARM 接收端并用合成批次验证，再安装生产 units，手工运行首次真实自动批次，最后 enable timers。
|
||||
- [x] 获批执行一次可控源端停止告警并恢复正常链路；不读取附件正文、不执行业务或企微其他写入。
|
||||
- [x] 删除个人工号默认值，新增生产/ARM 共用的 root-only 企微群机器人 helper；两端 systemd 只绑定 Webhook 文件路径。
|
||||
- [x] 增加 ARM 26 小时 dead-man service/timer，覆盖缺批次、超龄、损坏、重复降噪和恢复通知。
|
||||
- [x] 将 buildSha 不同降级为 finding；缺失/损坏/超龄/project 不同继续硬失败，并支持部署后立即运行一次备份。
|
||||
- [x] 为 backup/rsync/finalize 保存 `0600` 最近失败诊断，summary 与群消息只保留脱敏类别。
|
||||
- [x] 把同 UTC 日不同 run_id 从拒绝改为“完整校验后原子取代”，覆盖失败回滚和周检查点重建。
|
||||
- [x] 增加 Webhook 不进入 `.env.example`/仓库配置/报告的否定断言，并在 runbook 记录调整 `BACKUP_RUN_AT` 后需人工确认次日窗口。
|
||||
- [x] 本地门禁通过并完成只读复核后提交；不推送、不部署、不配置真实 Webhook。
|
||||
|
||||
## 预计文件边界
|
||||
|
||||
- 新增 `scripts/run-automatic-backup.sh`：生产 host 编排与 safe summary。
|
||||
- 新增 `scripts/receive-offhost-backup.sh`：ARM forced-command receiver、finalize 和 retention。
|
||||
- 新增 `deploy/trialrun/install-automatic-backup-timer.sh`：生产 units 安装/卸载。
|
||||
- 新增 `deploy/trialrun/install-offhost-backup-receiver.sh`：ARM rsync/账号/目录/公钥安装检查；软件安装和账号创建只在获批远端阶段执行。
|
||||
- 新增 `scripts/qa/run-automatic-backup-contract.sh`：纯合成正负向测试。
|
||||
- 修改 `server/src/scripts/send-wecom-ops-alert.js` 及其单测，仅做通用脱敏 summary 兼容。
|
||||
- 修改 `deploy/trialrun/validate-env-contract.js`、现有 runbook/checklist/管理员手册和部署规约。
|
||||
- 新增一个不含 Secret 的宿主告警 helper；只读取安装时传入的 root-only Webhook 文件。
|
||||
|
||||
## 验证命令
|
||||
|
||||
- `bash -n scripts/run-automatic-backup.sh scripts/receive-offhost-backup.sh deploy/trialrun/install-automatic-backup-timer.sh deploy/trialrun/install-offhost-backup-receiver.sh scripts/qa/run-automatic-backup-contract.sh`
|
||||
- `bash scripts/qa/run-automatic-backup-contract.sh`
|
||||
- `bash scripts/qa/run-backup-restore-drill.sh`
|
||||
- `cd server && node --check src/scripts/send-wecom-ops-alert.js`
|
||||
- `cd server && npm test -- --run tests/unit/scripts/send-wecom-ops-alert.test.js`
|
||||
- `node deploy/trialrun/validate-env-contract.js`
|
||||
- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-27-automatic-offhost-backup`
|
||||
- `node scripts/qa/check-trellis-governance.mjs`
|
||||
- `node scripts/qa/check-sensitive-content.mjs`
|
||||
- `node scripts/qa/audit-docs.mjs --check-registry`
|
||||
- `node scripts/qa/check-process-contract.mjs --baseline --json`
|
||||
- `git diff --check`
|
||||
- 合成测试必须模拟源端不再产生 finalize，并由 ARM dead-man 得到失败；不能只直接调用“发送消息”路径。
|
||||
|
||||
## 外部门禁与回滚点
|
||||
|
||||
- 安装 ARM rsync/账号/authorized_keys/目录属于远端配置写入，须单独批准；不得复用普通登录账号或本地工作站私钥。
|
||||
- 安装生产 SSH key/config/systemd units、首次真实备份、真实异机传输和企微失败告警均须单独批准。
|
||||
- 创建企微运维群机器人及把 Webhook 写入两台主机 root-only 文件由用户手工完成；Codex 不读取、不回显、不记录其内容。
|
||||
- 首次真实执行前记录现有手工批次数量、磁盘余量、活动容器/mount/build 和现有 timer；只输出脱敏计数/布尔值。
|
||||
- 任何目标身份、主机指纹、活动 mount、manifest 或哈希不匹配立即停止，不猜路径、不接受新指纹、不清理源端批次。
|
||||
- 回滚新 timer/receiver 时只撤销调度和专用授权；ARM archive、源端完整批次和原 WARCH3 timer 保留。
|
||||
|
||||
## 本地实施与验证证据
|
||||
|
||||
- 已实现生产 wrapper、ARM forced receiver、独立群机器人 helper、ARM dead-man 和两端安装器；生产 3 组 timer 与 ARM 1 组 timer 均默认只安装为 disabled。
|
||||
- `run-automatic-backup-contract.sh` 使用临时目录与 mock Docker/SSH/rsync 完成 46 项正负向合同，不连接真实主机；覆盖同日取代与回滚、坏候选保留、周检查点重建、Webhook 权限/链接/格式负例、build finding、状态损坏、源端停止、重复降噪和恢复通知。
|
||||
- 企微告警聚焦单测 15 项通过；stdin 路径被替换为安全类别，AUTO_BACKUP 首次失败/季度提醒发送且相同重复状态降噪。
|
||||
- `run-backup-restore-drill.sh` 本地合成 DB/附件往返通过，本次合成 RTO 为 8 秒；该结果不是生产 RTO 或 ARM 恢复证据。
|
||||
- `scripts/check-minimal.sh` 通过：server 162 个测试文件、1004 项测试全部通过，client typecheck/build 通过。
|
||||
- 环境合同、Trellis task/governance、敏感扫描、文档注册、流程合同和 `git diff --check` 均通过；文档注册保留既有 22 个未注册文件 warning，无本任务新增文档。
|
||||
- 最终只读复核修正了三个本地缺陷：生产安装器残留个人工号校验、ARM 重装可能改变 dead-man 状态文件属主、stale 状态未校验 run id/manifest SHA；修正后 46 项合同与 `scripts/check-minimal.sh` 全量复跑通过。
|
||||
- 更新后的 ARM dead-man/helper、生产 helper/runner、真实有效首批、四组 timer enable 与真实企微失败链路已在单独授权后完成；公司只有一名系统管理员，第二人季度走查不适用。
|
||||
|
||||
### 首次远端执行增量
|
||||
|
||||
- ARM 已安装 rsync、密码锁定专用账号和 forced-command receiver；从生产验证普通 shell、sender/read、delete 均拒绝，写通道 dry-run 通过且未生成归档。
|
||||
- 生产三组 units 已安装为 disabled/inactive，并绑定活动 Compose project、PGDATA、附件挂载、构建标识和固定 SSH 身份。
|
||||
- 首次真实备份曾在 `runtime_identity` 阶段因 Docker `println` 参数间空格 fail-closed；runner 改用 `docker inspect` mount JSON 后，后续真实源端成对批次已生成并传输。
|
||||
- ARM finalize 随后因同 UTC 日已有不同 run id 而拒绝，因此尚无可作为自动备份闭环证据的新批次；本轮已在本地实现“完整校验后同日取代”，更新后的远端脚本、首批和 timer enable 仍待另行批准。
|
||||
|
||||
### 远端启用与真实告警验收增量
|
||||
|
||||
- 获批部署 `a0006f0` 后，生产活动构建与预期一致；自动备份同日新批次经 ARM 完整校验后安全取代旧日批次，源端与目标端 manifest、文件大小和 SHA-256 一致，最近失败诊断不存在。
|
||||
- 生产 daily、watchdog、季度恢复提醒与 ARM dead-man 四组 timer 均为 `enabled/active`；最终只读复核中四个对应 service 最近结果均为 `success`、退出码均为 `0`。
|
||||
- daily 下一窗口为每日 02:30，ARM dead-man 为每日 06:30、26 小时阈值，生产 watchdog 为每日 07:10，季度恢复提醒为每季度首日 09:00；失败验收结束后 daily 已恢复正常调度。
|
||||
- 两端企微运维群机器人均由用户手工写入执行账号持有的 root-only `0600` 文件；Codex 未读取、回显或记录 Webhook 内容。
|
||||
- 真实失败链路通过停止源端 daily timer 并临时缩短 ARM 阈值触发：首次 dead-man 超龄只发送一条 `FAIL_OFFHOST_DEADMAN`/`archive_stale` 脱敏通知，相同失败重复执行明确降噪、不再次发送。
|
||||
- 恢复 daily、生成新的真实有效批次并恢复 ARM 正常阈值后，dead-man 返回 `success/0`，企微群只收到一条 `PASS_OFFHOST_DEADMAN_FRESH`/`healthy` 恢复通知；临时失败测试脚本已删除。
|
||||
- 本轮外部验收未读取附件正文、备份文件正文、原始日志或 Webhook 内容，未修改业务数据；恢复卡已保存在现有 runbook，由唯一系统管理员按季度提醒自行执行影子恢复。
|
||||
@@ -0,0 +1,76 @@
|
||||
# 每日自动异机备份
|
||||
|
||||
## Goal
|
||||
|
||||
为 5–10 人内部 OA 建立低维护的每日自动恢复点：生产每天生成同批数据库、附件和 manifest，完整校验后复制到 ARM 备份机，失败可被当天发现并告警；同时用简单保留策略和季度人工影子恢复降低主机故障与静默备份失效风险。
|
||||
|
||||
## Background
|
||||
|
||||
- 现有 `scripts/backup-all.sh` 已能生成 DB、附件和脱敏 manifest 的完整批次，并在发布完整目录前验证 dump、tar 和 SHA-256。
|
||||
- 当前仓库没有任何生产调度器调用 `backup-all.sh`；唯一活动 timer 是 07:30 运维巡检，不执行备份。
|
||||
- 生产主机和 ARM 均有 systemd、SSH、`flock` 和 SHA-256 工具；生产有 rsync，ARM 当前没有 rsync。
|
||||
- 两台主机之间当前没有可直接复用的免密 SSH 别名或密钥链路，不能把本地工作站的 SSH 配置误当作服务器间自动化能力。
|
||||
- 当前手工正确批次和 ARM 影子恢复已证明格式可用，但不能满足 RPO 24 小时或自动失败发现。
|
||||
- 已确认采用“生产主动推送到 ARM 的专用受限接收身份”：生产保留即时传输结果，ARM 的已归档批次不暴露给 rsync 写入根。
|
||||
- 任务实施前，生产切换已完成并应用 61 项 migration；本地为 62 项，`2526052` 后续部署只新增 1 项 migration。当时自动备份三组 units 已安装但尚未形成可靠启用闭环。
|
||||
- 首次真实自动批次曾证明源端成对备份可生成，但 ARM 因“同一 UTC 日仅允许一个不同 run_id”拒绝重试批次；当时即时告警仍依赖活动 server 容器，不能覆盖容器或整机故障。
|
||||
|
||||
## Scope
|
||||
|
||||
- 生产每日成对备份、ARM 受限接收与目标端校验。
|
||||
- 源端 7 个日批次、ARM 7 个日批次加 4 个周检查点的安全轮转。
|
||||
- 每日 systemd 调度、独立 stale watchdog、失败企微告警和季度人工恢复提醒。
|
||||
- 两端安装器、合成契约测试、现有运维文档/规约更新和一页脱敏恢复卡。
|
||||
|
||||
## Requirements
|
||||
|
||||
- 每日自动从唯一活动 Compose project 解析 DB 容器、活动附件挂载和实际构建标识，不读取旧项目、固定容器名或仓库默认附件目录。
|
||||
- 复用 `backup-all.sh` 生成完整批次；任何来源核验、dump、tar、manifest 或哈希检查失败时不得进入异机完成态。
|
||||
- 使用独立 systemd service/timer 和 `flock`，避免与部署、人工备份或自身上次运行重叠。
|
||||
- 完整批次通过固定主机指纹和专用受限密钥自动推送到 ARM；该身份无普通 shell，只能把文件写入 inbox 或请求校验/归档，不能通过 rsync 读取或删除 archive。
|
||||
- 传输端和目标端都要校验 manifest、文件大小和 SHA-256；同名已归档目标不得覆盖。中断传输可以在 inbox 内续传，但只有完整校验后才能原子进入 archive。
|
||||
- 生产保留最近 7 个成功日批次；ARM 至少保留最近 7 个日批次和 4 个周检查点。清理只能发生在本次异机成功确认之后，且不得删除最后一个已知成功批次。
|
||||
- 生产即时告警和 ARM dead-man 告警统一使用独立企微运维群机器人；Webhook 只从 root-only 文件读取，不依赖 OA server、数据库、员工绑定或默认个人工号，也不得进入仓库、`.env.example`、Trellis、日志或聊天。
|
||||
- ARM 安装独立 26 小时 dead-man timer，直接检查最近归档完成时间；生产整机或容器不可用、timer 未运行、传输未 finalize 时仍能从另一台主机发出告警。
|
||||
- 生产 watchdog 的“恢复点缺失/损坏/超龄/Compose project 不匹配”继续硬失败；仅 `buildSha` 早于当前部署时降级为一次 finding，不得误报为没有恢复点。部署后仍推荐立即触发一次获批备份。
|
||||
- backup、rsync 和 finalize 的失败输出必须写入状态目录下权限 `0600` 的最近失败诊断；企微告警仍只包含阶段、状态、finding 数和脱敏建议，不附原始输出。
|
||||
- 同一 UTC 日已有不同自动归档时，新批次必须先完整通过 manifest、文件集、大小和 SHA-256 校验，再以可回滚的同文件系统 rename 顺序原子取代旧日批次；任一替换步骤失败必须恢复旧批次或保留可诊断状态。最终仍只保留每日一个恢复点。
|
||||
- 告警只包含阶段、状态、finding 数和脱敏建议，不包含数据库凭证、完整目录、SSH 信息、真实业务数据、附件名或原始命令输出。
|
||||
- 季度恢复保持人工触发，在 ARM 隔离目标执行；季度 timer 只发送一次脱敏提醒，不得自动运行恢复，更不得触碰生产库。
|
||||
- 形成一页脱敏恢复卡，说明主机故障后的负责人、备份位置类别、恢复顺序和验收项，保存在现有运维 runbook 中，由唯一系统管理员维护并用于季度自查。
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] 本地合成测试证明每日 wrapper 只接受唯一活动来源、互斥锁有效，并只发布完整备份批次。
|
||||
- [x] 传输测试证明不完整批次、哈希不一致、重复批次和不可达 ARM 均失败，且不产生异机成功状态。
|
||||
- [x] 保留测试覆盖 7 个日批次、4 个周检查点、最后成功批次保护和仅在异机确认后清理。
|
||||
- [x] 告警测试证明首次失败和状态恶化会告警，重复同一失败降噪；07:10 stale 检查能发现超过 24 小时没有异机成功记录。
|
||||
- [x] 本地合成测试证明同日不同 run_id 的完整新批次可取代旧批次，损坏新批次不会影响旧归档，且周检查点与日归档不产生悬空或错误指向。
|
||||
- [x] 本地合成测试证明 buildSha 不同仅产生 finding，恢复点缺失/损坏/超龄/project 不同仍失败；失败诊断为 `0600` 且不进入告警摘要。
|
||||
- [x] 环境契约证明 Webhook 相关键和值不出现在 `.env.example`、仓库配置或报告；两端安装器只接受 root-only Webhook 文件路径。
|
||||
- [x] ARM 26 小时 dead-man 合成验证覆盖“源端停止/不产生新 finalize”而非仅验证机器人能发送，并覆盖首次失败、重复降噪与恢复通知。
|
||||
- [x] systemd 安装器显式绑定 Compose project、活动数据路径、构建标识、备份根和受限远端身份；安装后四组 timer 均 enabled/active，安装阶段未手工执行真实告警。
|
||||
- [x] 经单独生产授权后完成一次自动日备份：源端完整、ARM 目标端哈希一致、状态报告脱敏，且未读取附件正文或修改业务数据。
|
||||
- [x] 模拟源端停止产生一条脱敏企微告警，相同失败重复执行不再发送；恢复正常链路并完成成功批次后只发送一条恢复通知。
|
||||
- [x] 恢复卡已写入现有运维 runbook，覆盖恢复入口、影子目标和 `/ready`/关键计数验收项；公司只有一名系统管理员，不设置虚构的第二人走查门禁。
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 不引入备份平台、对象存储、MinIO、云厂商 SDK、Kubernetes 或持续复制。
|
||||
- 不自动执行生产恢复、数据库 reset、迁移改写或附件回写。
|
||||
- 不把真实业务正文、附件清单、凭证、主机地址、完整目录或 SSH 私钥写入仓库、报告或聊天。
|
||||
- 不把企微群机器人 Webhook URL、key 或配置文件正文写入仓库、`.env.example`、Trellis、报告、日志或聊天。
|
||||
- 不在本任务继续企微本人授权、员工角色、付款、报销或其他产品功能。
|
||||
- 不因自动备份完成而取消季度人工影子恢复。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Shell 语法与合成文件系统测试，覆盖锁、来源解析、传输失败、目标重复、哈希和保留轮转。
|
||||
- 合成 receiver 测试覆盖同日完整批次取代、损坏候选保留旧批次、替换失败恢复以及周检查点一致性。
|
||||
- 合成双主机告警测试覆盖生产无 server 容器、ARM 26 小时无新归档、重复降噪、恢复通知和 root-only Webhook 文件门禁。
|
||||
- 复用 `scripts/qa/run-backup-restore-drill.sh` 验证成对备份合同未回退。
|
||||
- 聚焦测试现有企微运维告警的备份状态与 stale finding 适配。
|
||||
- `node deploy/trialrun/validate-env-contract.js`。
|
||||
- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-27-automatic-offhost-backup`。
|
||||
- Trellis governance、敏感扫描、文档注册/流程合同和 `git diff --check`。
|
||||
- 生产安装、首次自动批次、真实失败告警和 ARM 目标验证分别保持外部门禁。
|
||||
@@ -0,0 +1,32 @@
|
||||
# 自动异机备份运行拓扑调查
|
||||
|
||||
## 边界
|
||||
|
||||
- 调查日期：2026-07-27。
|
||||
- 仅执行只读命令；未创建账号、密钥、目录、timer 或备份，未安装软件、传输数据或发送告警。
|
||||
- 只记录工具可用性、批次数量和连接布尔值，不记录主机地址、账号、完整路径、凭证或业务数据。
|
||||
|
||||
## 仓库事实
|
||||
|
||||
- `scripts/backup-all.sh` 只被手工文档和合成恢复演练调用，没有生产调度入口。
|
||||
- 完整批次由 `database.dump`、`storage.tar.gz` 和 `manifest.json` 组成；最终目录只在 dump/tar 可读取且 SHA-256 生成后发布。
|
||||
- DB Docker 路径要求显式容器和数据库参数；附件路径要求显式 `STORAGE_DIR_HOST`，不存在历史默认回退。
|
||||
- 07:30 WARCH3 timer 只运行每日巡检。现有告警 CLI 可读取通用 summary 的 `issueId`、`status`、`findingCount` 和报告目录，但帮助、状态文件和部分建议文案仍绑定 WARCH3，备份复用前需要小范围泛化或独立状态文件。
|
||||
|
||||
## 主机能力
|
||||
|
||||
| 能力 | 生产主机 | ARM 备份机 |
|
||||
| --- | --- | --- |
|
||||
| systemd / SSH / flock / SHA-256 | 可用 | 可用 |
|
||||
| rsync | 可用 | 当前不可用 |
|
||||
| 现成跨主机免密别名/链路 | 无 | 无 |
|
||||
|
||||
- 生产当前存在少量历史/手工批次，但没有定时策略。
|
||||
- 正确异机批次此前已复制并完成影子恢复；本轮默认目录探测没有把它当作新自动化根，说明自动化必须显式配置目标根，不能猜历史位置。
|
||||
|
||||
## 设计影响
|
||||
|
||||
- 无论推送或拉取，都需要一次性安装/配置 ARM rsync 并建立专用受限身份，不能复用本地工作站密钥作为生产事实。
|
||||
- 直接给生产保存 ARM 普通登录密钥虽然最省事，但会让生产失陷扩展为异机备份可删除，不应采用。
|
||||
- 生产推送更容易在同一任务内获得传输退出码并触发即时告警；要用无交互 shell、仅允许新批次写入的受限接收身份，并让已归档批次脱离该身份的删除权限。
|
||||
- ARM 拉取让生产主机没有 ARM 凭证，隔离更强；代价是调度和告警分散在两台机器，需要额外心跳或状态回传才能保证当天发现失败。
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"id": "automatic-offhost-backup",
|
||||
"name": "automatic-offhost-backup",
|
||||
"title": "每日自动异机备份",
|
||||
"description": "为内部 OA 建立每日成对备份、ARM 异机复制、保留轮转、失败告警与季度恢复提醒的低维护自动化基线。",
|
||||
"status": "completed",
|
||||
"dev_type": null,
|
||||
"scope": "scripts/backup automation; deploy/trialrun backup receiver and systemd installers; server ops-alert adapter/tests; deployment runbook/checklist/spec evidence",
|
||||
"package": null,
|
||||
"priority": "P0",
|
||||
"creator": "shunli",
|
||||
"assignee": "shunli",
|
||||
"createdAt": "2026-07-27",
|
||||
"completedAt": "2026-07-27",
|
||||
"branch": null,
|
||||
"base_branch": "main",
|
||||
"worktree_path": null,
|
||||
"commit": null,
|
||||
"pr_url": null,
|
||||
"subtasks": [],
|
||||
"children": [],
|
||||
"parent": null,
|
||||
"relatedFiles": [],
|
||||
"notes": "",
|
||||
"meta": {}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,75 @@
|
||||
# 异机备份最终收尾修复设计
|
||||
|
||||
## 边界与顺序
|
||||
|
||||
本任务只修三类已确认缺陷：凭证进入 argv、紧急恢复卡口径漂移、finalize 被强杀后的孤儿事务。receiver 的恢复入口固定为：校验配置 → 取得 `.finalize.lock` → 恢复或拒绝历史孤儿事务 → 校验本次批次 → 发布/取代 → 周检查点 → 保留轮转。没有锁时不得扫描或改变孤儿对象。
|
||||
|
||||
## 凭证传递合同
|
||||
|
||||
### 生产 wrapper
|
||||
|
||||
`capture_command` 仍负责将 stdout/stderr 写入 root-only 诊断，但不再把 `env` 当作被执行命令。调用方进入隔离子 shell，在子 shell 内导出 `DATABASE_URL`、Docker DB 身份、备份目录和构建标识，再调用：
|
||||
|
||||
```bash
|
||||
(
|
||||
export DATABASE_URL DB_TOOL_MODE DB_CONTAINER POSTGRES_DB POSTGRES_USER \
|
||||
POSTGRES_PASSWORD STORAGE_DIR_HOST BACKUP_DIR BACKUP_RUN_ID BUILD_SHA \
|
||||
SOURCE_IMAGE_DIGEST
|
||||
capture_command paired_backup "$ROOT_DIR/scripts/backup-all.sh"
|
||||
)
|
||||
```
|
||||
|
||||
敏感值仍只存在于执行进程环境和内存，不写入命令 argv、诊断、summary 或日志。本任务不宣称对 root 或同执行身份读取 `/proc/<pid>/environ` 提供额外隔离。
|
||||
|
||||
### Docker DB 工具
|
||||
|
||||
生产 Docker 已证明无值 `-e VAR` 能继承客户端环境。三处 Docker 调用统一为：
|
||||
|
||||
```bash
|
||||
PGPASSWORD="$POSTGRES_PASSWORD" docker exec [-i] -e PGPASSWORD \
|
||||
"$DB_CONTAINER" <pg-command-without-password>
|
||||
```
|
||||
|
||||
`PGPASSWORD` 的值不得出现在 Docker CLI 参数，mock 测试同时检查参数不含合成密码、容器命令能看到继承值。测试夹具用于启动命令的普通 `env KEY=value` 不在本轮机械改写范围。
|
||||
|
||||
## 日归档孤儿恢复
|
||||
|
||||
receiver 在锁内扫描 `.daily-replacement-YYYYMMDD.*`。每个孤儿必须是受控 archive 根的一层普通目录且非链接；从其 manifest 取得原 `runId`，执行与正式 finalize 相同的精确文件集、大小和 SHA-256 自校验，并验证日期与孤儿名一致。
|
||||
|
||||
| 现场 | 含义 | 正确动作 |
|
||||
| --- | --- | --- |
|
||||
| 孤儿存在且当日正式归档存在、两者均有效 | 新批次已换入，只缺最后清理 | 保留正式归档，删除孤儿 |
|
||||
| 孤儿存在且当日正式归档缺失、孤儿有效 | 旧归档已移开，新批次尚未换入 | 按孤儿 manifest 的 `runId` 原子恢复旧正式归档 |
|
||||
| 两者都在但当日正式归档校验不过 | 无法判断换入结果 | fail closed，两个都保留 |
|
||||
| 孤儿无效、同日多个正式归档、同日多个孤儿、名称/manifest 日期不符或任一对象为链接 | 状态歧义或受污染 | fail closed，所有对象保留 |
|
||||
|
||||
恢复旧归档或确认新归档后，才允许继续处理本次 partial。任何删除前再次验证目标仍是刚才检查过的严格路径和类型。
|
||||
|
||||
## 周检查点孤儿恢复
|
||||
|
||||
周检查点是日归档的硬链接派生物，但仍不得猜测或直接删除。每个 `week-YYYY-Www.new.*` / `.rollback.*` 先按正式周名分组，要求每类最多一个普通目录且非链接。正式周点和 rollback 必须自校验 manifest，且 manifest 日期属于该 ISO 周；`.new` 是从未提交的 staging，内容可能因强杀而不完整，不能作为恢复来源，也不要求内容有效，但只有在另一个有效权威来源已经确定后才可清理。
|
||||
|
||||
| 现场 | 含义 | 正确动作 |
|
||||
| --- | --- | --- |
|
||||
| rollback 存在、正式周点缺失、rollback 有效 | 正式周点已移开，新点尚未换入 | 原子恢复 rollback 为正式周点；`.new` 路径/类型安全时删除 staging，不使用其内容 |
|
||||
| rollback 存在、正式周点存在且正式点有效 | 新周点已换入，只缺清理 | 保留正式周点，删除 rollback；`.new` 路径/类型安全时删除 staging |
|
||||
| 仅 `.new` 存在、正式周点存在且有效 | staging 未提交或残留 | 保留正式周点；`.new` 路径/类型安全时删除 staging |
|
||||
| 仅 `.new` 存在、正式周点缺失 | 没有已提交周点 | 先找到该 ISO 周唯一有效的正式日归档来源；能唯一确定时删除 staging 并从该日归档重建，不能唯一确定时 fail closed 并保留 `.new` |
|
||||
| 正式周点无效、rollback 无效、多个候选、日期/周不符或任一对象为链接 | 状态歧义或受污染 | fail closed，全部保留 |
|
||||
|
||||
若日归档恢复动作影响某 ISO 周，必须从最终选定且已验证的正式日归档重建该周检查点后才算 reconciliation 成功。周点恢复/重建失败时不删除日归档 rollback。
|
||||
|
||||
## 失败传播与可观测性
|
||||
|
||||
reconciliation 任一歧义返回非零，receiver 不处理本次 partial。源端 SSH finalize 随即在 `offhost_finalize` 阶段失败，`capture_command` 保留 `0600` 诊断，`on_error` 通过独立企微群机器人发送脱敏类别，且不写 `last-success.json`、不清理源端批次。若故障持续，ARM dead-man 在超龄窗口独立告警。该双层链路已经真实验证，本任务只用合成测试证明返回值和源端状态传播，不再次制造真实断链。
|
||||
|
||||
## 文档与单管理员恢复卡
|
||||
|
||||
runbook 标题改为“紧急恢复卡（单管理员离线版）”，正文只保留唯一系统管理员、ARM 隔离影子恢复、真实生产恢复另行批准和不粘贴敏感内容的约束。复制到 ARM 与离线副本是后续可选运维动作，不写死真实路径、不进入代码验收。
|
||||
|
||||
## 兼容、部署与回滚
|
||||
|
||||
- 不改变批次、manifest、receiver forced-command、timer、保留数量或告警 payload 合同。
|
||||
- 本地提交后先停在外部门禁；推送、生产 runner/DB 脚本部署、ARM receiver 更新分别请求批准。
|
||||
- 代码回滚只回滚本任务提交；已存在的正式归档、周检查点、partial 和孤儿对象均不删除。
|
||||
- 完成后归档本任务并结束运维加固主线。
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,63 @@
|
||||
# 异机备份最终收尾修复实施计划
|
||||
|
||||
## 实施步骤
|
||||
|
||||
- [x] 先扩展合成测试：记录 mock Docker argv 与继承环境，证明四条凭证路径不含合成密码/完整连接串且容器能收到 `PGPASSWORD`。
|
||||
- [x] 修改 `run-automatic-backup.sh`，在隔离子 shell内导出成对备份变量后调用 `capture_command`，移除生产路径 `env KEY=value` argv。
|
||||
- [x] 修改 `backup-db.sh` 和 `restore-db.sh` 三处 Docker 调用，使用前缀环境赋值配合无值 `-e PGPASSWORD`；不批量改写测试夹具。
|
||||
- [x] 为 receiver 增加锁内 self-validation 与 orphan reconciliation helper，严格实现 `design.md` 的日归档和周检查点状态表。
|
||||
- [x] 增加孤儿测试矩阵：明确恢复、明确清理、周点重建、非法类型/链接/重复候选/校验失败全量保留，以及 fail-closed 到源端状态的合成传播。
|
||||
- [x] 更新 runbook 恢复卡标题和正文为单管理员口径；同步 `.trellis/spec/hyb-oa/deployment-trialrun.md` 的凭证 argv 与中断恢复稳定合同，不新建平行文档。
|
||||
- [x] 运行聚焦与全量相关门禁，完成只读差异复核；不连接真实告警、不写 ARM/生产、不执行 migration。
|
||||
- [x] 经用户确认后提交中文 commit；推送和生产/ARM 部署另行批准。
|
||||
- [x] 部署验证完成后归档本任务，硬收口运维主线并返回产品 backlog；不得因本轮审查再新增运维加固 task。
|
||||
|
||||
## 预计文件边界
|
||||
|
||||
- `scripts/run-automatic-backup.sh`
|
||||
- `scripts/backup-db.sh`
|
||||
- `scripts/restore-db.sh`
|
||||
- `scripts/receive-offhost-backup.sh`
|
||||
- `scripts/qa/run-automatic-backup-contract.sh`
|
||||
- 必要时扩展现有 `scripts/qa/run-backup-restore-drill.sh` 的 argv 断言，不新增平行 runner。
|
||||
- `deploy/trialrun/runbook.md`
|
||||
- `.trellis/spec/hyb-oa/deployment-trialrun.md`
|
||||
- 当前 Trellis task 三份规划/证据文件。
|
||||
|
||||
## 验证命令
|
||||
|
||||
- `bash -n scripts/run-automatic-backup.sh scripts/backup-db.sh scripts/restore-db.sh scripts/receive-offhost-backup.sh scripts/qa/run-automatic-backup-contract.sh scripts/qa/run-backup-restore-drill.sh`
|
||||
- `bash scripts/qa/run-automatic-backup-contract.sh`
|
||||
- `bash scripts/qa/run-backup-restore-drill.sh`
|
||||
- `node deploy/trialrun/validate-env-contract.js`
|
||||
- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-27-offhost-backup-final-fixes`
|
||||
- `node scripts/qa/check-trellis-governance.mjs`
|
||||
- `node scripts/qa/check-sensitive-content.mjs`
|
||||
- `node scripts/qa/audit-docs.mjs --check-registry`
|
||||
- `node scripts/qa/check-process-contract.mjs --baseline --json`
|
||||
- `git diff --check`
|
||||
|
||||
## 外部门禁与回滚点
|
||||
|
||||
- 生产 Docker 无值 `-e VAR` 继承只读实测已完成；实施阶段不重复读取生产配置或凭证。
|
||||
- 推送当前 3 个未推送提交与本任务提交，需要提交完成后另行批准。
|
||||
- 更新生产自动备份脚本、ARM receiver、手工触发真实备份或复制恢复卡到 ARM，均需分别明确批准。
|
||||
- 生产部署前只读确认活动构建、62 项 migration、四组 timer 和最近成功状态；本任务没有 migration。
|
||||
- 任一 orphan 合成负例发生对象减少、正式恢复点丢失或歧义状态被自动选择，立即停止，不进入部署。
|
||||
- 回滚只回退本任务脚本/文档，不清理现场备份、partial、rollback 或已归档 task。
|
||||
|
||||
## 本地实施与验证证据（2026-07-27）
|
||||
|
||||
- 凭证传递已覆盖 wrapper、nightly Docker 备份、Docker 影子确认和恢复；mock 捕获的 argv 不含合成连接串/口令，容器命令仍收到继承的 `PGPASSWORD`。
|
||||
- receiver 合同共 73 项通过。除原有同日取代、保留轮转、dead-man 和失败传播外，新增日/周孤儿恢复、重复/链接/损坏歧义 fail closed，以及“周 reconciliation 失败时日 rollback 不提前删除”的回归断言。
|
||||
- `run-backup-restore-drill.sh` 通过，隔离合成 RTO 为 9 秒；`validate-env-contract.js` 返回 `[R26][PASS]`。
|
||||
- shell 语法、task validate、Trellis governance、敏感内容扫描、文档注册审计、流程 baseline 和 `git diff --check` 均通过。文档审计仅报告仓库既有 22 个未注册 docs 文件警告，本任务未新增 docs 文件。
|
||||
- 只读复核发现并修复了“先删日 rollback、后重建周点”的顺序缺陷；稳定规则已同步到 `deployment-trialrun.md`。本地过程未连接真实企微、ARM 或生产，也未执行 migration。
|
||||
|
||||
## 受控远端更新证据（2026-07-27）
|
||||
|
||||
- `01f1259` 与此前三个收尾提交已推送到 `origin/main`，本地与远端分支一致。
|
||||
- 经单独批准，生产端只原子替换自动备份 runner、数据库备份和影子恢复三份脚本；远端 SHA-256 与提交内容一致，shell 语法通过。三组自动备份 timer 保持 enabled/active，对应 service 均为 inactive/success；未手工启动备份。
|
||||
- 经单独批准，ARM staging receiver 更新后由管理员通过既有 root 安装 wrapper 安装。安装器内部受限账户配置校验通过；正式 receiver 与提交内容 SHA-256 一致，属主/权限为 root:root `0755`，dead-man timer 保持 enabled/active，service 为 inactive/success。
|
||||
- 本轮远端更新未触发真实备份、企微告警、数据库 migration 或业务数据写入；公网业务构建和 62 项 migration 基线不在本次脚本更新范围内。
|
||||
- 至此本任务的代码、推送和远端验证均完成；按既定边界归档后结束运维加固主线，后续返回产品 backlog。
|
||||
@@ -0,0 +1,62 @@
|
||||
# 异机备份最终收尾修复
|
||||
|
||||
## Goal
|
||||
|
||||
在不扩建运维平台的前提下，完成自动异机备份最后一轮有界修复：让生产热路径和 Docker 备份/恢复入口不再把数据库凭证放入进程 argv；让紧急恢复卡与唯一系统管理员的事实一致；让 ARM finalize 在被强杀后能够安全恢复明确的孤儿事务，并在歧义状态下保留全部现场、失败告警而非静默停摆。
|
||||
|
||||
## Background
|
||||
|
||||
- `origin/main` 与当前生产活动构建均为 `a0006f0`；`2526052` 已推送，生产已有 62 项成功 migration，手机号 nullable migration 不需要重复部署。
|
||||
- 当前待推送提交只有 `de35ab8`、`95b51e6`、`f1e8cf1`；本任务完成前不单独推送，避免本地与远端继续形成多个收尾批次。
|
||||
- `scripts/run-automatic-backup.sh:344` 使用 `env KEY=value` 调用成对备份，完整 `DATABASE_URL` 和数据库口令会进入 `env` 进程 argv。
|
||||
- `scripts/backup-db.sh:74`、`scripts/restore-db.sh:101`、`scripts/restore-db.sh:113` 使用 `docker exec -e PGPASSWORD=value`，其中 nightly 自动备份会经过 `backup-db.sh`，因此只修 wrapper 不能关闭热路径暴露。
|
||||
- 生产 Docker `20.10.24` 已用无敏感合成变量只读实测：`VAR=value docker exec -e VAR ...` 能把客户端环境值继承给容器命令；测试只记录 `inherited=yes`，未读取或输出数据库凭证。
|
||||
- `deploy/trialrun/runbook.md` 的紧急恢复卡仍写“复制到公司知识库”和“第二名内部人员”，与已经批准的唯一系统管理员口径矛盾。
|
||||
- receiver 已有 `flock` 和正常错误回滚，但 `SIGKILL` 无法执行 shell 清理；强杀窗口可能留下 `.daily-replacement-*`、`.new.*` 或 `.rollback.*`，下一次 finalize 当前没有启动恢复逻辑。
|
||||
|
||||
## Scope
|
||||
|
||||
- 修复 wrapper 与三处 Docker DB 工具调用的凭证 argv 暴露，并增加合成回归断言。
|
||||
- 增加 receiver 锁内日归档/周检查点孤儿事务恢复及 fail-closed 测试矩阵。
|
||||
- 同步单管理员紧急恢复卡和部署规约；完成本地验证、提交与后续受控部署后归档运维主线。
|
||||
|
||||
## Requirements
|
||||
|
||||
- 四处已确认的凭证调用必须改为环境继承，敏感值不得出现在调用命令 argv：wrapper 在隔离子 shell 中导出所需变量后调用 `capture_command`；Docker 入口使用 `PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD ...`，不得使用 `env DATABASE_URL=...` 或 `-e PGPASSWORD=...`。
|
||||
- 契约测试中的 `env KEY=value` 是合成夹具，不含真实凭证，不做无意义的批量改写；新增断言必须使用合成标记证明敏感值不出现在 mock 捕获的 argv。
|
||||
- 紧急恢复卡标题、负责人和季度动作必须统一为单管理员口径：恢复卡保存在现有 runbook，由唯一系统管理员维护并按季度在 ARM 隔离目标执行影子恢复；不虚构知识库副本或第二人门禁。
|
||||
- finalize 必须在取得现有独占锁之后、处理本次 partial 之前，扫描严格命名的日归档和周检查点孤儿事务；正式点和 rollback 只有在普通目录、非符号链接、位于受控根目录且通过 manifest/文件集/大小/SHA-256 自校验后才可作为权威来源。`.new` 永远不是权威来源，只能在严格路径/类型校验且另一个有效权威来源已经确定后清理。
|
||||
- `.daily-replacement-*` 的判定必须遵守 `design.md` 的日归档状态表：孤儿与有效当日归档同时存在时删除孤儿；孤儿存在但当日归档缺失时从孤儿 manifest 还原原归档名；当日归档校验失败、存在多个候选或任何身份歧义时 fail closed，两个对象均保留。
|
||||
- 周检查点 `.new.*`/`.rollback.*` 必须遵守 `design.md` 的对称状态表：rollback 在正式周检查点缺失时优先恢复；正式周检查点有效时临时对象才可清理；只有 staging 且正式点缺失时，必须先找到同 ISO 周唯一、有效的日归档来源再重建；歧义或校验失败一律保留现场并失败。
|
||||
- 自动清理必须逐目标执行严格路径、类型和命名校验；不得用宽泛 glob 删除，不得触碰正式日归档、正式周检查点、手工批次或最后一个有效恢复点。
|
||||
- orphan 歧义导致 receiver 非零时，源端必须继续在 `offhost_finalize` 阶段失败、保留源批次和 `0600` 诊断并走独立群机器人告警；ARM dead-man 继续在超龄窗口独立告警。本任务复用已真实验证的链路，不再人为制造第二次真实断链。
|
||||
- 恢复卡复制到 ARM 和离线副本属于可选纯运维缓解项，不是代码验收阻断；任何 ARM 写入、推送、生产部署或真实备份执行仍需另行批准。
|
||||
- 本任务完成后硬收口运维加固，不继续新增备份平台、告警能力、保留层级或恢复自动化，后续返回产品 backlog。
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] 合成 argv 测试证明 wrapper、nightly Docker 备份、Docker 影子确认和 Docker 影子恢复四条路径的命令参数均不包含合成密码或完整连接串，同时容器命令仍收到 `PGPASSWORD`。
|
||||
- [x] `run-automatic-backup.sh` 不再出现生产路径 `capture_command ... env DATABASE_URL=...`；`backup-db.sh`/`restore-db.sh` 不再出现 `docker exec ... -e PGPASSWORD=value`。
|
||||
- [x] 恢复卡正文、标题、负责人和季度动作与《管理员手册》单管理员口径一致，且仍保留真实生产恢复的单独批准门禁。
|
||||
- [x] 日归档测试覆盖“孤儿+有效正式点则清理孤儿”“孤儿+缺正式点则恢复旧点”“孤儿+无效正式点则两个都保留并失败”“重复/非法/链接候选则失败且不删除”。
|
||||
- [x] 周检查点测试覆盖 `.new`、`.rollback`、正式点存在/缺失、可唯一重建和歧义失败；任何成功恢复后只保留一个正式周检查点，任何歧义失败后现场对象不减少。
|
||||
- [x] orphan fail closed 通过本地合成链路证明 finalize 非零、源端不写成功状态、不执行源端保留清理且诊断为 `0600`；不重复执行真实企微断链验收。
|
||||
- [x] 现有同日取代、失败回滚、7 日 + 4 周保留、dead-man、备份/恢复往返、环境合同和敏感扫描均无回退。
|
||||
- [x] task、Trellis governance、文档注册/流程合同和 `git diff --check` 通过；只有经确认的任务文件进入提交。
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 不新增数据库 migration，不重复部署 `2526052`，不修改生产业务数据。
|
||||
- 不重测已经通过的真实企微失败、重复降噪和恢复通知链路。
|
||||
- 不自动执行恢复，不删除 ARM 正式归档或手工批次，不改变 7 日 + 4 周保留策略。
|
||||
- 不改写仅含合成值的测试夹具 `env` 调用，不做全仓无关安全重构。
|
||||
- 不在本任务直接推送、部署、更新 ARM 脚本或复制恢复卡；这些动作分别保持外部门禁。
|
||||
- 不继续新增运维能力；本任务之后不因本轮审查再派生新的运维加固 task。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Shell 语法检查和 `bash scripts/qa/run-automatic-backup-contract.sh`，扩展 mock argv/环境继承与 orphan 恢复矩阵。
|
||||
- `bash scripts/qa/run-backup-restore-drill.sh`，覆盖 Docker 备份与影子恢复入口无回退。
|
||||
- `node deploy/trialrun/validate-env-contract.js`。
|
||||
- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-27-offhost-backup-final-fixes`。
|
||||
- Trellis governance、敏感扫描、文档注册、流程合同和 `git diff --check`。
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"id": "offhost-backup-final-fixes",
|
||||
"name": "offhost-backup-final-fixes",
|
||||
"title": "异机备份最终收尾修复",
|
||||
"description": "修复自动异机备份凭证进入进程 argv、单管理员恢复卡漂移和 finalize 强杀后孤儿事务恢复，完成后硬收口运维主线。",
|
||||
"status": "completed",
|
||||
"dev_type": null,
|
||||
"scope": "scripts/run-automatic-backup.sh; scripts/backup-db.sh; scripts/restore-db.sh; scripts/receive-offhost-backup.sh; automatic-backup contracts; trialrun runbook/spec",
|
||||
"package": null,
|
||||
"priority": "P0",
|
||||
"creator": "shunli",
|
||||
"assignee": "shunli",
|
||||
"createdAt": "2026-07-27",
|
||||
"completedAt": "2026-07-27",
|
||||
"branch": null,
|
||||
"base_branch": "main",
|
||||
"worktree_path": null,
|
||||
"commit": null,
|
||||
"pr_url": null,
|
||||
"subtasks": [],
|
||||
"children": [],
|
||||
"parent": null,
|
||||
"relatedFiles": [],
|
||||
"notes": "",
|
||||
"meta": {}
|
||||
}
|
||||
@@ -91,7 +91,7 @@ export type EmployeeSummary = {
|
||||
id: string;
|
||||
employeeNo: string;
|
||||
name: string;
|
||||
phone: string;
|
||||
phone: string | null;
|
||||
email: string;
|
||||
status: EmployeeStatus;
|
||||
departmentId: string;
|
||||
@@ -231,7 +231,7 @@ export type EmployeeProfile = {
|
||||
id: string;
|
||||
employeeNo: string;
|
||||
name: string;
|
||||
phone: string;
|
||||
phone: string | null;
|
||||
email: string;
|
||||
status: EmployeeStatus;
|
||||
employmentStatus: EmployeeEmploymentStatus;
|
||||
|
||||
+1
-1
@@ -35,7 +35,7 @@ export type MeResponse = {
|
||||
id: string;
|
||||
employeeNo: string;
|
||||
name: string;
|
||||
phone: string;
|
||||
phone: string | null;
|
||||
email: string;
|
||||
departmentId: string;
|
||||
directManagerId: string | null;
|
||||
|
||||
+2
-2
@@ -6,7 +6,7 @@ export type UserSummary = {
|
||||
id: string;
|
||||
employeeNo: string;
|
||||
name: string;
|
||||
phone: string;
|
||||
phone: string | null;
|
||||
email: string;
|
||||
status: UserStatus;
|
||||
departmentId: string;
|
||||
@@ -89,7 +89,7 @@ export async function fetchUsers(params: {
|
||||
|
||||
export async function createUser(payload: {
|
||||
name: string;
|
||||
phone: string;
|
||||
phone: string | null;
|
||||
email: string;
|
||||
departmentId?: string;
|
||||
directManagerId?: string | null;
|
||||
|
||||
@@ -802,7 +802,7 @@ export function EmployeeProfilePage() {
|
||||
<div>
|
||||
<div className="text-sm text-zinc-500">手机 / 邮箱</div>
|
||||
<div className="mt-2 text-sm text-zinc-900">
|
||||
{profile.employee.phone} / {profile.employee.email}
|
||||
{profile.employee.phone ?? "未填写"} / {profile.employee.email}
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
|
||||
@@ -322,7 +322,7 @@ export function UserManagementPage() {
|
||||
</TableCell>
|
||||
<TableCell className="text-sm">
|
||||
<div>{item.email}</div>
|
||||
<div className="text-zinc-500">{item.phone}</div>
|
||||
<div className="text-zinc-500">{item.phone ?? "手机号未填写"}</div>
|
||||
</TableCell>
|
||||
<TableCell><StatusBadge status={item.status} /></TableCell>
|
||||
<TableCell className="text-xs">{formatRoleCodes(item.roleCodes)}</TableCell>
|
||||
|
||||
@@ -13,7 +13,7 @@ import { Label } from "../../components/ui/label";
|
||||
|
||||
export interface CreateUserFormData {
|
||||
name: string;
|
||||
phone: string;
|
||||
phone: string | null;
|
||||
email: string;
|
||||
departmentId: string | undefined;
|
||||
directManagerId: string | null;
|
||||
@@ -61,7 +61,6 @@ export function CreateUserDialog({
|
||||
const canSubmit =
|
||||
!isPending &&
|
||||
trimmedName !== "" &&
|
||||
trimmedPhone !== "" &&
|
||||
trimmedEmail !== "" &&
|
||||
rolesArray.length >= 1;
|
||||
|
||||
@@ -84,7 +83,7 @@ export function CreateUserDialog({
|
||||
function handleSubmit() {
|
||||
onSubmit({
|
||||
name: trimmedName,
|
||||
phone: trimmedPhone,
|
||||
phone: trimmedPhone || null,
|
||||
email: trimmedEmail,
|
||||
departmentId: departmentId || undefined,
|
||||
directManagerId: directManagerId ? directManagerId : null,
|
||||
@@ -117,13 +116,13 @@ export function CreateUserDialog({
|
||||
/>
|
||||
</div>
|
||||
<div className="grid gap-2">
|
||||
<Label htmlFor="phone">手机号</Label>
|
||||
<Label htmlFor="phone">手机号（可暂不填写）</Label>
|
||||
<Input
|
||||
id="phone"
|
||||
data-testid="users-create-phone"
|
||||
value={phone}
|
||||
onChange={(e) => setPhone(e.target.value)}
|
||||
placeholder="13800000001"
|
||||
placeholder="后续可由本人授权或管理员补齐"
|
||||
/>
|
||||
</div>
|
||||
<div className="grid gap-2">
|
||||
|
||||
@@ -31,7 +31,7 @@ export interface UserImportDialogProps {
|
||||
const TEMPLATE_CSV = [
|
||||
"employeeNo,name,phone,email,departmentName,title,directManagerEmployeeNo,roles,hireDate",
|
||||
"E1001,李经理,13800000001,[email protected],行政部,经理,,manager,2026-03-01",
|
||||
"E1002,张三,13800000002,[email protected],行政部,员工,E1001,employee,2026-03-15",
|
||||
"E1002,张三,,[email protected],行政部,员工,E1001,employee,2026-03-15",
|
||||
].join("\n");
|
||||
|
||||
function toCsvCell(value: string) {
|
||||
@@ -205,10 +205,10 @@ export function UserImportDialog({ open, onOpenChange, onImported }: UserImportD
|
||||
}}
|
||||
/>
|
||||
<div className="text-xs text-zinc-500">
|
||||
必填列：employeeNo、name、phone、email、departmentName
|
||||
必填列：employeeNo、name、email、departmentName
|
||||
</div>
|
||||
<div className="text-xs text-zinc-500">
|
||||
选填列：title、directManagerEmployeeNo、roles、hireDate
|
||||
选填列：phone、title、directManagerEmployeeNo、roles、hireDate
|
||||
</div>
|
||||
{selectedFile ? (
|
||||
<div className="rounded-lg border border-zinc-200 px-3 py-2 text-xs text-zinc-600">
|
||||
|
||||
+11
-1
@@ -31,7 +31,7 @@
|
||||
- [ ] 正式 deploy 使用绝对数据路径；既有实例使用 `DEPLOY_DATA_MODE=existing`，PGDATA 已存在且含 `PG_VERSION`，附件目录已存在
|
||||
- [ ] 已证明没有两个 PostgreSQL 实例同时挂载同一 PGDATA；任何不一致均停止发布
|
||||
- [ ] 已记录活动 server/db 容器、不可变镜像 ID（及可用 repo digest）、Caddy 目标、挂载源、环境摘要，不记录 env 正文
|
||||
- [ ] 已冻结成功 migration 数量（当前预期 58）和关键业务表逐表计数基线
|
||||
- [ ] 已现场只读冻结成功 migration 数量和关键业务表逐表计数基线；当前生产证据为 61 项、本地为 62 项，本次待上线只应新增 1 项，现场结果不符立即停止
|
||||
- [ ] 确认当前版本已打 tag：`git tag -a trialrun-YYYYMMDD-NN -m "trialrun release"`
|
||||
- [ ] 本地门禁：`./scripts/check-minimal.sh`
|
||||
- [ ] 部署契约校验：`node deploy/trialrun/validate-env-contract.js`
|
||||
@@ -47,6 +47,16 @@
|
||||
- [ ] 同一批次包含 `database.dump`、`storage.tar.gz`、`manifest.json`，两份 SHA-256 校验通过
|
||||
- [ ] 完整批次已复制到获批异机/异盘，并记录保留周期
|
||||
- [ ] 同批次已在隔离影子目标恢复：migration 数、关键业务表逐表计数、附件普通文件数均与冻结基线一致，兼容镜像 `/ready` 通过
|
||||
- [ ] 自动异机备份（未完成远端门禁前保持未勾选）：
|
||||
- [ ] **EXTERNAL_GATED**：用户已在独立企微运维群创建机器人，并分别把 Webhook 写入生产、ARM 的 root-only `0600` 文件；未向 Codex、仓库、普通 `.env`、日志或聊天提供正文
|
||||
- [ ] **EXTERNAL_GATED**：ARM 已获批安装 rsync、密码锁定专用账号、forced-command key、独立 inbox/archive/weekly 和 26 小时 dead-man；普通 shell、rsync sender/read/delete 均被拒绝
|
||||
- [ ] **EXTERNAL_GATED**：生产专用私钥为 `0600`，known-hosts 指纹已人工核对，安装器显式绑定活动 project、PGDATA、附件、完整/短构建 SHA、备份/状态目录、远端身份和 Webhook 文件路径
|
||||
- [ ] `install-automatic-backup-timer.sh install` 只安装 disabled units；首批真实批次及 ARM manifest/SHA-256 核对通过后才显式执行 `enable`
|
||||
- [ ] 生产 daily/watchdog/drill-reminder 与 ARM dead-man 四组 timer enabled/active，各 oneshot service 当前 inactive；未通过只调用机器人发送入口来伪造验证
|
||||
- [ ] 最近一次异机成功不超过 24 小时；生产自动日批次为 7 个，ARM 自动日批次为 7 个、周检查点为 4 个，历史手工批次未被轮转
|
||||
- [ ] **EXTERNAL_GATED**：已真正停止一次源端 timer 或故意破坏一次传输，ARM dead-man 仅产生一条脱敏企微告警；恢复 finalize 后收到一次恢复通知，同一失败重复运行已降噪
|
||||
- [ ] 若调整过 `BACKUP_RUN_AT`，已人工确认调整后的次日 UTC 分桶和 26 小时窗口，未把一次预期空窗误判为长期故障
|
||||
- [ ] 本季度已由第二名内部人员按 runbook 首部恢复卡在 ARM 隔离目标完成影子恢复走查；timer 未自动恢复生产
|
||||
- [ ] 停机复制方案已冻结：旧 PostgreSQL 完全停止后复制到空新目录并保留所有权/权限/链接/扩展属性；旧目录原位保留且观察期内不再写入
|
||||
|
||||
## B. 发布（Deploy）
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
REMOTE_DIR="${REMOTE_DIR:-/opt/hyb-oa-trialrun}"
|
||||
REMOTE_ENV_FILE="${REMOTE_ENV_FILE:-${REMOTE_DIR}/deploy/trialrun/.env}"
|
||||
COMPOSE_FILE="${COMPOSE_FILE:-deploy/trialrun/docker-compose.yml}"
|
||||
COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-}"
|
||||
POSTGRES_DATA_DIR="${POSTGRES_DATA_DIR:-}"
|
||||
STORAGE_HOST_DIR="${STORAGE_HOST_DIR:-}"
|
||||
BUILD_SHA="${BUILD_SHA:-}"
|
||||
VITE_BUILD_SHA="${VITE_BUILD_SHA:-}"
|
||||
AUTOMATIC_BACKUP_DIR="${AUTOMATIC_BACKUP_DIR:-}"
|
||||
AUTOMATIC_BACKUP_STATE_DIR="${AUTOMATIC_BACKUP_STATE_DIR:-}"
|
||||
OFFHOST_TARGET="${OFFHOST_TARGET:-}"
|
||||
OFFHOST_SSH_KEY="${OFFHOST_SSH_KEY:-}"
|
||||
OFFHOST_KNOWN_HOSTS="${OFFHOST_KNOWN_HOSTS:-}"
|
||||
ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
|
||||
ALERT_HELPER="${ALERT_HELPER:-${REMOTE_DIR}/scripts/send-wecom-group-alert.py}"
|
||||
BACKUP_RUN_AT="${BACKUP_RUN_AT:-02:30}"
|
||||
WATCHDOG_RUN_AT="${WATCHDOG_RUN_AT:-07:10}"
|
||||
SERVICE_PREFIX="${SERVICE_PREFIX:-hyb-oa-automatic-backup}"
|
||||
MODE="${1:-install}"
|
||||
|
||||
RUNNER="${REMOTE_DIR}/scripts/run-automatic-backup.sh"
|
||||
SYSTEMD_DIR="/etc/systemd/system"
|
||||
|
||||
fail() {
|
||||
echo "[automatic-backup-installer][ERROR] $1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
need_root() {
|
||||
[[ "$(id -u)" == "0" ]] || fail "root privileges are required"
|
||||
}
|
||||
|
||||
validate_absolute_path() {
|
||||
local name="$1"
|
||||
local value="$2"
|
||||
if [[ -z "$value" || "$value" != /* || ! "$value" =~ ^/[A-Za-z0-9._/-]+$ ]]; then
|
||||
fail "$name must be an explicit safe absolute path"
|
||||
fi
|
||||
if [[ "$value" == "/" || "$value" == */ || "$value" == *"//"* \
|
||||
|| "/$value/" == *"/../"* || "/$value/" == *"/./"* ]]; then
|
||||
fail "$name contains an unsafe path segment"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_managed_directory_path() {
|
||||
local name="$1"
|
||||
local value="$2"
|
||||
validate_absolute_path "$name" "$value"
|
||||
[[ "$value" =~ ^/[^/]+/[^/]+ ]] \
|
||||
|| fail "$name must be a dedicated path below a top-level directory"
|
||||
case "$value" in
|
||||
/bin/*|/boot/*|/dev/*|/etc/*|/lib/*|/lib64/*|/proc/*|/run/*|/sbin/*|/sys/*|/usr/*)
|
||||
fail "$name must not be inside a system-managed directory"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
paths_overlap() {
|
||||
local first="$1"
|
||||
local second="$2"
|
||||
[[ "$first" == "$second" || "$first" == "$second"/* || "$second" == "$first"/* ]]
|
||||
}
|
||||
|
||||
validate_inputs() {
|
||||
local name value
|
||||
for name in REMOTE_DIR REMOTE_ENV_FILE POSTGRES_DATA_DIR STORAGE_HOST_DIR \
|
||||
AUTOMATIC_BACKUP_DIR AUTOMATIC_BACKUP_STATE_DIR OFFHOST_SSH_KEY OFFHOST_KNOWN_HOSTS \
|
||||
ALERT_WEBHOOK_FILE ALERT_HELPER; do
|
||||
value="${!name:-}"
|
||||
validate_absolute_path "$name" "$value"
|
||||
done
|
||||
validate_managed_directory_path AUTOMATIC_BACKUP_DIR "$AUTOMATIC_BACKUP_DIR"
|
||||
validate_managed_directory_path AUTOMATIC_BACKUP_STATE_DIR "$AUTOMATIC_BACKUP_STATE_DIR"
|
||||
[[ -d "$REMOTE_DIR" && -f "$REMOTE_ENV_FILE" && -x "$RUNNER" ]] \
|
||||
|| fail "deployment root, env file, or backup runner is missing"
|
||||
[[ -f "$POSTGRES_DATA_DIR/PG_VERSION" && -d "$STORAGE_HOST_DIR" ]] \
|
||||
|| fail "active data directories are invalid"
|
||||
[[ "$COMPOSE_PROJECT_NAME" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "Compose project name is invalid"
|
||||
[[ "$COMPOSE_FILE" =~ ^[A-Za-z0-9._/-]+$ && "$COMPOSE_FILE" != /* \
|
||||
&& "/$COMPOSE_FILE/" != *"/../"* && "/$COMPOSE_FILE/" != *"/./"* ]] \
|
||||
|| fail "Compose file must be a safe relative path"
|
||||
[[ "$BUILD_SHA" =~ ^[0-9a-fA-F]{40,64}$ ]] || fail "full build SHA is invalid"
|
||||
BUILD_SHA="${BUILD_SHA,,}"
|
||||
VITE_BUILD_SHA="${VITE_BUILD_SHA,,}"
|
||||
[[ "$VITE_BUILD_SHA" =~ ^[0-9a-f]{12}$ && "${BUILD_SHA:0:12}" == "$VITE_BUILD_SHA" ]] \
|
||||
|| fail "frontend build SHA must match the full build prefix"
|
||||
[[ "$OFFHOST_TARGET" =~ ^[a-z_][a-z0-9_-]*@[A-Za-z0-9._-]+$ ]] \
|
||||
|| fail "off-host target must use a safe user@alias value"
|
||||
[[ -f "$OFFHOST_SSH_KEY" && ! -L "$OFFHOST_SSH_KEY" \
|
||||
&& "$(stat -c %a "$OFFHOST_SSH_KEY")" == "600" ]] \
|
||||
|| fail "off-host private key must exist with mode 0600"
|
||||
[[ -s "$OFFHOST_KNOWN_HOSTS" ]] || fail "pinned known-hosts file is missing"
|
||||
[[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
|
||||
&& "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" \
|
||||
&& "$(stat -c %u "$ALERT_WEBHOOK_FILE")" == "0" ]] \
|
||||
|| fail "alert webhook file must be root-owned with mode 0600"
|
||||
[[ -f "$ALERT_HELPER" && ! -L "$ALERT_HELPER" ]] || fail "alert helper is missing"
|
||||
[[ "$BACKUP_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "backup time is invalid"
|
||||
[[ "$WATCHDOG_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "watchdog time is invalid"
|
||||
[[ "$SERVICE_PREFIX" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "service prefix is invalid"
|
||||
local -a isolated_paths=(
|
||||
"$POSTGRES_DATA_DIR" "$STORAGE_HOST_DIR" "$AUTOMATIC_BACKUP_DIR" "$AUTOMATIC_BACKUP_STATE_DIR"
|
||||
)
|
||||
local first_index second_index
|
||||
for ((first_index = 0; first_index < ${#isolated_paths[@]}; first_index += 1)); do
|
||||
for ((second_index = first_index + 1; second_index < ${#isolated_paths[@]}; second_index += 1)); do
|
||||
if paths_overlap "${isolated_paths[$first_index]}" "${isolated_paths[$second_index]}"; then
|
||||
fail "data, backup, and state directories must be independent"
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
environment_lines() {
|
||||
cat <<ENVIRONMENT
|
||||
Environment=REMOTE_DIR=${REMOTE_DIR}
|
||||
Environment=REMOTE_ENV_FILE=${REMOTE_ENV_FILE}
|
||||
Environment=COMPOSE_FILE=${COMPOSE_FILE}
|
||||
Environment=COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME}
|
||||
Environment=POSTGRES_DATA_DIR=${POSTGRES_DATA_DIR}
|
||||
Environment=STORAGE_HOST_DIR=${STORAGE_HOST_DIR}
|
||||
Environment=BUILD_SHA=${BUILD_SHA}
|
||||
Environment=VITE_BUILD_SHA=${VITE_BUILD_SHA}
|
||||
Environment=AUTOMATIC_BACKUP_DIR=${AUTOMATIC_BACKUP_DIR}
|
||||
Environment=AUTOMATIC_BACKUP_STATE_DIR=${AUTOMATIC_BACKUP_STATE_DIR}
|
||||
Environment=OFFHOST_TARGET=${OFFHOST_TARGET}
|
||||
Environment=OFFHOST_SSH_KEY=${OFFHOST_SSH_KEY}
|
||||
Environment=OFFHOST_KNOWN_HOSTS=${OFFHOST_KNOWN_HOSTS}
|
||||
Environment=ALERT_WEBHOOK_FILE=${ALERT_WEBHOOK_FILE}
|
||||
Environment=ALERT_HELPER=${ALERT_HELPER}
|
||||
ENVIRONMENT
|
||||
}
|
||||
|
||||
write_service() {
|
||||
local suffix="$1"
|
||||
local mode="$2"
|
||||
local description="$3"
|
||||
local service_file="$SYSTEMD_DIR/${SERVICE_PREFIX}-${suffix}.service"
|
||||
cat >"$service_file" <<SERVICE
|
||||
[Unit]
|
||||
Description=${description}
|
||||
After=docker.service network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=${REMOTE_DIR}
|
||||
UMask=0077
|
||||
$(environment_lines)
|
||||
ExecStart=/bin/bash ${RUNNER} ${mode}
|
||||
SERVICE
|
||||
}
|
||||
|
||||
write_timer() {
|
||||
local suffix="$1"
|
||||
local calendar="$2"
|
||||
local description="$3"
|
||||
local timer_file="$SYSTEMD_DIR/${SERVICE_PREFIX}-${suffix}.timer"
|
||||
cat >"$timer_file" <<TIMER
|
||||
[Unit]
|
||||
Description=${description}
|
||||
|
||||
[Timer]
|
||||
OnCalendar=${calendar}
|
||||
Persistent=true
|
||||
Unit=${SERVICE_PREFIX}-${suffix}.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
TIMER
|
||||
}
|
||||
|
||||
install_units() {
|
||||
need_root
|
||||
validate_inputs
|
||||
mkdir -p "$AUTOMATIC_BACKUP_DIR" "$AUTOMATIC_BACKUP_STATE_DIR"
|
||||
[[ -d "$AUTOMATIC_BACKUP_DIR" && ! -L "$AUTOMATIC_BACKUP_DIR" \
|
||||
&& -d "$AUTOMATIC_BACKUP_STATE_DIR" && ! -L "$AUTOMATIC_BACKUP_STATE_DIR" ]] \
|
||||
|| fail "backup and state directories must not be symbolic links"
|
||||
chmod 0700 "$AUTOMATIC_BACKUP_DIR" "$AUTOMATIC_BACKUP_STATE_DIR"
|
||||
|
||||
write_service daily run "hyb-oa automatic paired off-host backup"
|
||||
write_service watchdog check-stale "hyb-oa automatic backup freshness watchdog"
|
||||
write_service drill-reminder remind-drill "hyb-oa quarterly shadow restore reminder"
|
||||
write_timer daily "*-*-* ${BACKUP_RUN_AT}:00" "Run hyb-oa automatic off-host backup daily"
|
||||
write_timer watchdog "*-*-* ${WATCHDOG_RUN_AT}:00" "Check hyb-oa off-host backup freshness daily"
|
||||
write_timer drill-reminder "*-01,04,07,10-01 09:00:00" "Remind quarterly hyb-oa shadow restore drill"
|
||||
systemctl daemon-reload
|
||||
echo "[automatic-backup-installer][PASS] units installed disabled"
|
||||
}
|
||||
|
||||
enable_units() {
|
||||
install_units
|
||||
systemctl enable --now \
|
||||
"${SERVICE_PREFIX}-daily.timer" \
|
||||
"${SERVICE_PREFIX}-watchdog.timer" \
|
||||
"${SERVICE_PREFIX}-drill-reminder.timer"
|
||||
echo "[automatic-backup-installer][PASS] timers enabled"
|
||||
}
|
||||
|
||||
uninstall_units() {
|
||||
need_root
|
||||
[[ "$SERVICE_PREFIX" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "service prefix is invalid"
|
||||
local suffix
|
||||
for suffix in daily watchdog drill-reminder; do
|
||||
systemctl disable --now "${SERVICE_PREFIX}-${suffix}.timer" >/dev/null 2>&1 || true
|
||||
systemctl stop "${SERVICE_PREFIX}-${suffix}.service" >/dev/null 2>&1 || true
|
||||
rm -f -- "$SYSTEMD_DIR/${SERVICE_PREFIX}-${suffix}.timer" \
|
||||
"$SYSTEMD_DIR/${SERVICE_PREFIX}-${suffix}.service"
|
||||
done
|
||||
systemctl daemon-reload
|
||||
echo "[automatic-backup-installer][PASS] units removed; backups and credentials retained"
|
||||
}
|
||||
|
||||
case "$MODE" in
|
||||
install)
|
||||
install_units
|
||||
;;
|
||||
enable)
|
||||
enable_units
|
||||
;;
|
||||
uninstall|--uninstall)
|
||||
uninstall_units
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: <explicit deployment/data/backup/SSH env> bash install-automatic-backup-timer.sh [install|enable|uninstall]"
|
||||
;;
|
||||
*)
|
||||
fail "unsupported mode"
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,235 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BACKUP_USER="${BACKUP_USER:-hybbackup}"
|
||||
BACKUP_ROOT="${BACKUP_ROOT:-}"
|
||||
PUBLIC_KEY_FILE="${PUBLIC_KEY_FILE:-}"
|
||||
RECEIVER_SOURCE="${RECEIVER_SOURCE:-}"
|
||||
RECEIVER_TARGET="${RECEIVER_TARGET:-/usr/local/sbin/hyb-oa-offhost-receiver}"
|
||||
CONFIG_FILE="${CONFIG_FILE:-/etc/hyb-oa-offhost-backup.conf}"
|
||||
ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
|
||||
ALERT_HELPER_SOURCE="${ALERT_HELPER_SOURCE:-}"
|
||||
ALERT_HELPER_TARGET="${ALERT_HELPER_TARGET:-/usr/local/sbin/hyb-oa-wecom-group-alert}"
|
||||
DEADMAN_SOURCE="${DEADMAN_SOURCE:-}"
|
||||
DEADMAN_TARGET="${DEADMAN_TARGET:-/usr/local/sbin/hyb-oa-offhost-deadman}"
|
||||
DEADMAN_MAX_AGE_SECONDS="${DEADMAN_MAX_AGE_SECONDS:-93600}"
|
||||
DEADMAN_RUN_AT="${DEADMAN_RUN_AT:-06:30}"
|
||||
DEADMAN_SERVICE="${DEADMAN_SERVICE:-hyb-oa-offhost-deadman}"
|
||||
SYSTEMD_DIR="${SYSTEMD_DIR:-/etc/systemd/system}"
|
||||
DAILY_KEEP="${DAILY_KEEP:-7}"
|
||||
WEEKLY_KEEP="${WEEKLY_KEEP:-4}"
|
||||
MODE="${1:-install}"
|
||||
|
||||
fail() {
|
||||
echo "[offhost-receiver-installer][ERROR] $1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
need_root() {
|
||||
[[ "$(id -u)" == "0" ]] || fail "root privileges are required"
|
||||
}
|
||||
|
||||
validate_absolute_path() {
|
||||
local name="$1"
|
||||
local value="$2"
|
||||
if [[ -z "$value" || "$value" != /* || ! "$value" =~ ^/[A-Za-z0-9._/-]+$ ]]; then
|
||||
fail "$name must be an explicit safe absolute path"
|
||||
fi
|
||||
if [[ "$value" == "/" || "$value" == */ || "$value" == *"//"* \
|
||||
|| "/$value/" == *"/../"* || "/$value/" == *"/./"* ]]; then
|
||||
fail "$name contains an unsafe path segment"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_backup_root() {
|
||||
validate_absolute_path BACKUP_ROOT "$BACKUP_ROOT"
|
||||
[[ "$BACKUP_ROOT" =~ ^/[^/]+/[^/]+ ]] \
|
||||
|| fail "backup root must be a dedicated path below a top-level directory"
|
||||
case "$BACKUP_ROOT" in
|
||||
/bin/*|/boot/*|/dev/*|/etc/*|/lib/*|/lib64/*|/proc/*|/run/*|/sbin/*|/sys/*|/usr/*)
|
||||
fail "backup root must not be inside a system-managed directory"
|
||||
;;
|
||||
esac
|
||||
[[ ! -L "$BACKUP_ROOT" ]] || fail "backup root must not be a symbolic link"
|
||||
if [[ -d "$BACKUP_ROOT" ]]; then
|
||||
local unmanaged_entry
|
||||
unmanaged_entry="$(find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 \
|
||||
! -name inbox ! -name archive ! -name weekly ! -name state -print -quit)"
|
||||
[[ -z "$unmanaged_entry" ]] || fail "backup root contains an unmanaged top-level entry"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_inputs() {
|
||||
[[ "$BACKUP_USER" =~ ^[a-z_][a-z0-9_-]{0,30}$ ]] || fail "backup user is invalid"
|
||||
validate_backup_root
|
||||
validate_absolute_path PUBLIC_KEY_FILE "$PUBLIC_KEY_FILE"
|
||||
validate_absolute_path RECEIVER_SOURCE "$RECEIVER_SOURCE"
|
||||
validate_absolute_path RECEIVER_TARGET "$RECEIVER_TARGET"
|
||||
validate_absolute_path CONFIG_FILE "$CONFIG_FILE"
|
||||
validate_absolute_path ALERT_WEBHOOK_FILE "$ALERT_WEBHOOK_FILE"
|
||||
validate_absolute_path ALERT_HELPER_SOURCE "$ALERT_HELPER_SOURCE"
|
||||
validate_absolute_path ALERT_HELPER_TARGET "$ALERT_HELPER_TARGET"
|
||||
validate_absolute_path DEADMAN_SOURCE "$DEADMAN_SOURCE"
|
||||
validate_absolute_path DEADMAN_TARGET "$DEADMAN_TARGET"
|
||||
[[ -f "$PUBLIC_KEY_FILE" && ! -L "$PUBLIC_KEY_FILE" ]] || fail "public key file is missing"
|
||||
[[ -f "$RECEIVER_SOURCE" && ! -L "$RECEIVER_SOURCE" ]] || fail "receiver source is missing"
|
||||
[[ -f "$ALERT_HELPER_SOURCE" && ! -L "$ALERT_HELPER_SOURCE" ]] || fail "alert helper source is missing"
|
||||
[[ -f "$DEADMAN_SOURCE" && ! -L "$DEADMAN_SOURCE" ]] || fail "dead-man source is missing"
|
||||
[[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
|
||||
&& "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" \
|
||||
&& "$(stat -c %u "$ALERT_WEBHOOK_FILE")" == "0" ]] \
|
||||
|| fail "alert webhook file must be root-owned with mode 0600"
|
||||
[[ "$DEADMAN_MAX_AGE_SECONDS" =~ ^[1-9][0-9]*$ ]] || fail "dead-man threshold must be positive"
|
||||
[[ "$DEADMAN_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "dead-man time is invalid"
|
||||
[[ "$DEADMAN_SERVICE" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "dead-man service name is invalid"
|
||||
[[ "$DAILY_KEEP" =~ ^[1-9][0-9]*$ && "$WEEKLY_KEEP" =~ ^[1-9][0-9]*$ ]] \
|
||||
|| fail "retention values must be positive"
|
||||
command -v rsync >/dev/null 2>&1 || fail "rsync must be installed before receiver setup"
|
||||
command -v rrsync >/dev/null 2>&1 || fail "rrsync must be installed before receiver setup"
|
||||
command -v python3 >/dev/null 2>&1 || fail "python3 is required"
|
||||
}
|
||||
|
||||
read_public_key() {
|
||||
local key_type key_body remainder extra_line_count
|
||||
extra_line_count="$(awk 'NF { count += 1 } END { print count + 0 }' "$PUBLIC_KEY_FILE")"
|
||||
[[ "$extra_line_count" == "1" ]] || fail "public key file must contain exactly one non-empty key"
|
||||
read -r key_type key_body remainder <"$PUBLIC_KEY_FILE"
|
||||
[[ "$key_type" == "ssh-ed25519" || "$key_type" == "[email protected]" ]] \
|
||||
|| fail "only Ed25519 public keys are accepted"
|
||||
[[ "$key_body" =~ ^[A-Za-z0-9+/=]+$ ]] || fail "public key body is invalid"
|
||||
printf '%s %s' "$key_type" "$key_body"
|
||||
}
|
||||
|
||||
install_receiver() {
|
||||
need_root
|
||||
validate_inputs
|
||||
|
||||
if ! id "$BACKUP_USER" >/dev/null 2>&1; then
|
||||
useradd --create-home --shell /bin/bash "$BACKUP_USER"
|
||||
fi
|
||||
passwd -l "$BACKUP_USER" >/dev/null 2>&1 || true
|
||||
|
||||
local home_dir inbox_dir archive_dir weekly_dir state_dir ssh_dir auth_file public_key rrsync_bin
|
||||
home_dir="$(getent passwd "$BACKUP_USER" | awk -F: '{print $6}')"
|
||||
validate_absolute_path BACKUP_USER_HOME "$home_dir"
|
||||
inbox_dir="$BACKUP_ROOT/inbox"
|
||||
archive_dir="$BACKUP_ROOT/archive"
|
||||
weekly_dir="$BACKUP_ROOT/weekly"
|
||||
state_dir="$BACKUP_ROOT/state"
|
||||
mkdir -p "$inbox_dir" "$archive_dir" "$weekly_dir" "$state_dir" "$home_dir/.ssh"
|
||||
[[ -d "$BACKUP_ROOT" && ! -L "$BACKUP_ROOT" \
|
||||
&& -d "$inbox_dir" && ! -L "$inbox_dir" \
|
||||
&& -d "$archive_dir" && ! -L "$archive_dir" \
|
||||
&& -d "$weekly_dir" && ! -L "$weekly_dir" ]] \
|
||||
|| fail "backup receiver directories must not be symbolic links"
|
||||
chown "$BACKUP_USER:$BACKUP_USER" "$BACKUP_ROOT"
|
||||
chown -R "$BACKUP_USER:$BACKUP_USER" "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh"
|
||||
chmod 0700 "$BACKUP_ROOT" "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh"
|
||||
chown -R root:root "$state_dir"
|
||||
chmod 0700 "$state_dir"
|
||||
|
||||
install -o root -g root -m 0755 "$RECEIVER_SOURCE" "$RECEIVER_TARGET"
|
||||
install -o root -g root -m 0755 "$ALERT_HELPER_SOURCE" "$ALERT_HELPER_TARGET"
|
||||
install -o root -g root -m 0755 "$DEADMAN_SOURCE" "$DEADMAN_TARGET"
|
||||
rrsync_bin="$(command -v rrsync)"
|
||||
cat >"$CONFIG_FILE" <<CONFIG
|
||||
BACKUP_INBOX_DIR=${inbox_dir}
|
||||
BACKUP_ARCHIVE_DIR=${archive_dir}
|
||||
BACKUP_WEEKLY_DIR=${weekly_dir}
|
||||
RRSYNC_BIN=${rrsync_bin}
|
||||
PYTHON_BIN=python3
|
||||
DAILY_KEEP=${DAILY_KEEP}
|
||||
WEEKLY_KEEP=${WEEKLY_KEEP}
|
||||
CONFIG
|
||||
chown root:root "$CONFIG_FILE"
|
||||
chmod 0644 "$CONFIG_FILE"
|
||||
|
||||
ssh_dir="$home_dir/.ssh"
|
||||
auth_file="$ssh_dir/authorized_keys"
|
||||
if [[ -s "$auth_file" ]] && ! grep -Fq 'hyb-oa-managed-offhost-backup' "$auth_file"; then
|
||||
fail "dedicated backup account already has unmanaged authorized keys"
|
||||
fi
|
||||
public_key="$(read_public_key)"
|
||||
cat >"$auth_file" <<AUTH
|
||||
# hyb-oa-managed-offhost-backup
|
||||
command="${RECEIVER_TARGET}",no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-pty,no-user-rc ${public_key}
|
||||
AUTH
|
||||
chown "$BACKUP_USER:$BACKUP_USER" "$auth_file"
|
||||
chmod 0600 "$auth_file"
|
||||
|
||||
runuser -u "$BACKUP_USER" -- env RECEIVER_CONFIG_FILE="$CONFIG_FILE" \
|
||||
"$RECEIVER_TARGET" validate-config >/dev/null
|
||||
cat >"$SYSTEMD_DIR/${DEADMAN_SERVICE}.service" <<SERVICE
|
||||
[Unit]
|
||||
Description=hyb-oa independent off-host backup dead-man
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
UMask=0077
|
||||
ExecStart=/usr/bin/python3 ${DEADMAN_TARGET} --archive-root ${archive_dir} --state-dir ${state_dir} --webhook-file ${ALERT_WEBHOOK_FILE} --alert-helper ${ALERT_HELPER_TARGET} --max-age-seconds ${DEADMAN_MAX_AGE_SECONDS}
|
||||
SERVICE
|
||||
cat >"$SYSTEMD_DIR/${DEADMAN_SERVICE}.timer" <<TIMER
|
||||
[Unit]
|
||||
Description=Check hyb-oa off-host backup freshness independently
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* ${DEADMAN_RUN_AT}:00
|
||||
Persistent=true
|
||||
Unit=${DEADMAN_SERVICE}.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
TIMER
|
||||
systemctl daemon-reload
|
||||
echo "[offhost-receiver-installer][PASS] restricted receiver and disabled dead-man installed"
|
||||
}
|
||||
|
||||
enable_receiver() {
|
||||
install_receiver
|
||||
systemctl enable --now "${DEADMAN_SERVICE}.timer"
|
||||
echo "[offhost-receiver-installer][PASS] dead-man timer enabled"
|
||||
}
|
||||
|
||||
uninstall_receiver() {
|
||||
need_root
|
||||
[[ "$BACKUP_USER" =~ ^[a-z_][a-z0-9_-]{0,30}$ ]] || fail "backup user is invalid"
|
||||
validate_absolute_path RECEIVER_TARGET "$RECEIVER_TARGET"
|
||||
validate_absolute_path CONFIG_FILE "$CONFIG_FILE"
|
||||
validate_absolute_path ALERT_HELPER_TARGET "$ALERT_HELPER_TARGET"
|
||||
validate_absolute_path DEADMAN_TARGET "$DEADMAN_TARGET"
|
||||
[[ "$DEADMAN_SERVICE" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "dead-man service name is invalid"
|
||||
systemctl disable --now "${DEADMAN_SERVICE}.timer" >/dev/null 2>&1 || true
|
||||
systemctl stop "${DEADMAN_SERVICE}.service" >/dev/null 2>&1 || true
|
||||
rm -f -- "$SYSTEMD_DIR/${DEADMAN_SERVICE}.timer" "$SYSTEMD_DIR/${DEADMAN_SERVICE}.service"
|
||||
if id "$BACKUP_USER" >/dev/null 2>&1; then
|
||||
local home_dir auth_file
|
||||
home_dir="$(getent passwd "$BACKUP_USER" | awk -F: '{print $6}')"
|
||||
auth_file="$home_dir/.ssh/authorized_keys"
|
||||
if [[ -f "$auth_file" ]] && grep -Fq 'hyb-oa-managed-offhost-backup' "$auth_file"; then
|
||||
rm -f -- "$auth_file"
|
||||
fi
|
||||
fi
|
||||
rm -f -- "$RECEIVER_TARGET" "$CONFIG_FILE" "$ALERT_HELPER_TARGET" "$DEADMAN_TARGET"
|
||||
systemctl daemon-reload
|
||||
echo "[offhost-receiver-installer][PASS] receiver authorization and dead-man removed; archives retained"
|
||||
}
|
||||
|
||||
case "$MODE" in
|
||||
install)
|
||||
install_receiver
|
||||
;;
|
||||
enable)
|
||||
enable_receiver
|
||||
;;
|
||||
uninstall|--uninstall)
|
||||
uninstall_receiver
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: <explicit backup/key/helper/dead-man/webhook paths> bash install-offhost-backup-receiver.sh [install|enable|uninstall]"
|
||||
;;
|
||||
*)
|
||||
fail "unsupported mode"
|
||||
;;
|
||||
esac
|
||||
@@ -5,6 +5,10 @@ REMOTE_DIR="${REMOTE_DIR:-/opt/hyb-oa-trialrun}"
|
||||
REMOTE_ENV_FILE="${REMOTE_ENV_FILE:-${REMOTE_DIR}/deploy/trialrun/.env}"
|
||||
COMPOSE_FILE="${COMPOSE_FILE:-deploy/trialrun/docker-compose.yml}"
|
||||
COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-hyb-oa-trialrun}"
|
||||
POSTGRES_DATA_DIR="${POSTGRES_DATA_DIR:-}"
|
||||
STORAGE_HOST_DIR="${STORAGE_HOST_DIR:-}"
|
||||
BUILD_SHA="${BUILD_SHA:-}"
|
||||
VITE_BUILD_SHA="${VITE_BUILD_SHA:-}"
|
||||
SERVICE_NAME="${SERVICE_NAME:-hyb-oa-warch3-ops}"
|
||||
RUN_AT="${RUN_AT:-07:30}"
|
||||
MODE="${1:-install}"
|
||||
@@ -19,6 +23,20 @@ need_root() {
|
||||
fi
|
||||
}
|
||||
|
||||
validate_data_path() {
|
||||
local name="$1"
|
||||
local value="$2"
|
||||
if [[ -z "$value" || "$value" != /* || ! "$value" =~ ^/[A-Za-z0-9._/-]+$ ]]; then
|
||||
echo "[ERROR] $name 必须显式提供受限字符集的绝对路径" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ "$value" == "/" || "$value" == */ || "$value" == *"//"* \
|
||||
|| "/$value/" == *"/../"* || "/$value/" == *"/./"* ]]; then
|
||||
echo "[ERROR] $name 不得是根目录、尾随斜杠或包含空、.、.. 路径段" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
uninstall_timer() {
|
||||
need_root
|
||||
systemctl disable --now "${SERVICE_NAME}.timer" >/dev/null 2>&1 || true
|
||||
@@ -42,6 +60,32 @@ install_timer() {
|
||||
echo "[ERROR] COMPOSE_PROJECT_NAME 格式无效" >&2
|
||||
exit 1
|
||||
fi
|
||||
validate_data_path POSTGRES_DATA_DIR "$POSTGRES_DATA_DIR"
|
||||
validate_data_path STORAGE_HOST_DIR "$STORAGE_HOST_DIR"
|
||||
if [[ "$POSTGRES_DATA_DIR" == "$STORAGE_HOST_DIR" \
|
||||
|| "$POSTGRES_DATA_DIR" == "$STORAGE_HOST_DIR"/* \
|
||||
|| "$STORAGE_HOST_DIR" == "$POSTGRES_DATA_DIR"/* ]]; then
|
||||
echo "[ERROR] PostgreSQL 与附件宿主目录必须相互独立" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -f "$POSTGRES_DATA_DIR/PG_VERSION" ]]; then
|
||||
echo "[ERROR] POSTGRES_DATA_DIR 必须是包含 PG_VERSION 的现存 PGDATA" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -d "$STORAGE_HOST_DIR" ]]; then
|
||||
echo "[ERROR] STORAGE_HOST_DIR 必须是现存目录" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! "$BUILD_SHA" =~ ^[0-9a-fA-F]{40,64}$ ]]; then
|
||||
echo "[ERROR] BUILD_SHA 必须是 40-64 位十六进制完整提交标识" >&2
|
||||
exit 1
|
||||
fi
|
||||
BUILD_SHA="${BUILD_SHA,,}"
|
||||
VITE_BUILD_SHA="${VITE_BUILD_SHA,,}"
|
||||
if [[ ! "$VITE_BUILD_SHA" =~ ^[0-9a-f]{12}$ || "${BUILD_SHA:0:12}" != "$VITE_BUILD_SHA" ]]; then
|
||||
echo "[ERROR] VITE_BUILD_SHA 必须是 BUILD_SHA 的 12 位十六进制前缀" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
local compose_exec
|
||||
if docker compose version >/dev/null 2>&1; then
|
||||
@@ -66,6 +110,10 @@ Environment=REMOTE_DIR=${REMOTE_DIR}
|
||||
Environment=REMOTE_ENV_FILE=${REMOTE_ENV_FILE}
|
||||
Environment=COMPOSE_FILE=${COMPOSE_FILE}
|
||||
Environment=COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME}
|
||||
Environment=POSTGRES_DATA_DIR=${POSTGRES_DATA_DIR}
|
||||
Environment=STORAGE_HOST_DIR=${STORAGE_HOST_DIR}
|
||||
Environment=BUILD_SHA=${BUILD_SHA}
|
||||
Environment=VITE_BUILD_SHA=${VITE_BUILD_SHA}
|
||||
Environment=LIVE_SMOKE_BASE_URL=https://oa.hyinda.com
|
||||
Environment=WARCH3_SEND_ALERT=1
|
||||
Environment=WARCH3_OPS_ALERT_SEND=1
|
||||
@@ -99,7 +147,7 @@ case "$MODE" in
|
||||
uninstall_timer
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: REMOTE_DIR=/opt/hyb-oa-trialrun bash deploy/trialrun/install-warch3-ops-timer.sh [install|uninstall]"
|
||||
echo "Usage: REMOTE_DIR=/opt/hyb-oa-trialrun COMPOSE_PROJECT_NAME=<project> POSTGRES_DATA_DIR=<absolute-pgdata> STORAGE_HOST_DIR=<absolute-storage> BUILD_SHA=<full-sha> VITE_BUILD_SHA=<12-char-prefix> bash deploy/trialrun/install-warch3-ops-timer.sh [install|uninstall]"
|
||||
;;
|
||||
*)
|
||||
echo "[ERROR] unsupported mode: $MODE" >&2
|
||||
|
||||
+64
-1
@@ -2,6 +2,27 @@
|
||||
|
||||
> 目标：按本文档在一台新机器完成 **启动 → 迁移 → 初始化管理员 → 验收 → 备份 → 恢复 → 回滚** 演练。
|
||||
|
||||
## 紧急恢复卡（单管理员离线版）
|
||||
|
||||
适用情形：生产主机不可恢复、数据库或附件损坏，且已由负责人批准进入事故恢复。公司当前只有一名系统管理员，由该管理员维护并执行本卡；不得在聊天或工单中粘贴凭证、业务正文、真实目录或原始日志。
|
||||
|
||||
1. 先暂停外部入口和新业务写入，不删除故障主机、旧数据目录或 ARM 归档；记录故障时间、最近成功批次编号类别和恢复负责人。
|
||||
2. 在 ARM 的只读归档中选择最近一个 `complete` 批次，复核三文件、manifest、SHA-256 与来源标识；先复制到隔离恢复工作区，不直接对归档原件操作。
|
||||
3. 在获批的新主机/隔离目标依次执行以下三个入口；尖括号值只从受控运维配置取得，不写回本文：
|
||||
|
||||
```bash
|
||||
RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE=<shadow-db> <explicit-db-env> \
|
||||
./scripts/restore-db.sh <verified-batch>/database.dump
|
||||
RESTORE_SCOPE=shadow ./scripts/restore-storage.sh \
|
||||
<verified-batch>/storage.tar.gz <empty-shadow-storage>
|
||||
curl -fsS <shadow-ready-url>
|
||||
```
|
||||
|
||||
4. 复核 `/ready`、运行构建标识、成功 migration 数、组织/权限/报销/付款等关键表计数、附件普通文件数及 DB/附件关联；任何一项不一致都停止切换。
|
||||
5. 真实生产恢复仍是 **EXTERNAL_GATED**：另建或更新获批 Trellis task，冻结新数据目录、回滚点和停机窗口后才切换。恢复后保留 ARM 归档和故障主机原数据，记录 RTO/RPO 与结论。
|
||||
|
||||
唯一系统管理员必须每季度按本卡在 ARM 隔离目标完成人工影子恢复；自动 timer 只提醒，不自动恢复。可在 ARM 和离线介质各保留一份脱敏副本，但不得写入凭证、真实目录或原始日志。
|
||||
|
||||
## 0A. 如果官网 `hyb-web` 已在同机运行（推荐方案）
|
||||
- 官网继续保持 `hyinda.com / www.hyinda.com` 静态托管，不要改官网目录和配置。
|
||||
- OA 新增独立二级域名，例如 `oa.hyinda.com`。
|
||||
@@ -205,6 +226,48 @@ manifest 必须至少记录一个真实来源标识：活动 `/ready` 有构建
|
||||
|
||||
整批复制到获批的异机/异盘位置并记录保留周期；不能只复制 DB 或附件中的一份。
|
||||
|
||||
#### 每日自动异机备份（可靠启用闭环仍为 EXTERNAL_GATED）
|
||||
|
||||
- 每日 02:30 由生产主机解析唯一活动 Compose project 和真实 DB/附件挂载，生成同批三文件，ARM 校验并原子归档成功后才写成功状态和执行源端轮转。
|
||||
- 生产保留 7 个自动日批次；ARM 保留 7 个日批次和 4 个周检查点。07:10 源端 watchdog 检查最近 24 小时异机成功记录；ARM 独立 dead-man 检查最近 26 小时归档，生产整机或 OA server 停止时仍能告警；季度 timer 只发送人工影子恢复提醒。
|
||||
- ARM 使用专用密码锁定账号和 forced-command key：rsync 只能写 inbox，不能读取/删除 archive，也没有普通 shell。生产固定专用 key 和人工核对的 known-hosts，禁止 `accept-new`。
|
||||
- 同一 UTC 日重试使用新 run id 时，ARM 必须先完整校验候选，再原子取代当日旧批次并重建该周硬链接检查点；坏候选或发布失败不得触碰原有效批次。
|
||||
- 生产与 ARM 都直接使用独立企微运维群机器人，不依赖 OA server、数据库或个人工号。Webhook 正文只由用户手工写入两台主机的 root-only `0600` 文件；仓库、普通 `.env`、systemd unit、日志和聊天只能出现受控文件路径，不能出现 URL 或 key。
|
||||
- 失败告警只包含阶段、状态、finding 数和类别建议；原始命令诊断只保留在源端状态目录的 `0600` 最近失败日志。同一失败降噪；首次失败、状态恶化、新 finding、恢复通知和每季度首次恢复提醒才发送。新部署构建与最近恢复点 build SHA 不同只记 finding，缺失、损坏、超龄、未来时间或 Compose project 不同仍硬失败。
|
||||
|
||||
创建机器人后，由用户分别登录两台主机，在 root shell 中用静默输入把完整 Webhook 写入受控路径；不要把值作为命令参数、环境文件内容或聊天消息交给 Codex：
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
read -rsp 'WeCom operations robot Webhook: ' HYB_OA_ALERT_WEBHOOK
|
||||
printf '\n'
|
||||
install -o root -g root -m 0600 /dev/null <controlled-webhook-file>
|
||||
printf '%s\n' "$HYB_OA_ALERT_WEBHOOK" > <controlled-webhook-file>
|
||||
unset HYB_OA_ALERT_WEBHOOK
|
||||
```
|
||||
|
||||
安装顺序必须是：先获批 ARM 端安装 rsync、受限身份和 receiver，用合成批次验证；再获批生产端安装 units，手工完成首个真实批次；最后显式执行生产安装器的 `enable`。两个安装器默认 `install` 均不代表已启用，不得把本地合同通过写成远端完成。
|
||||
|
||||
```bash
|
||||
# ARM：参数从受控配置取得；install 会写账号/authorized_keys/receiver，需单独批准。
|
||||
BACKUP_ROOT=<absolute-backup-root> PUBLIC_KEY_FILE=<dedicated-public-key> \
|
||||
RECEIVER_SOURCE=<repo-receiver-script> ALERT_WEBHOOK_FILE=<controlled-webhook-file> \
|
||||
ALERT_HELPER_SOURCE=<repo-group-alert-helper> DEADMAN_SOURCE=<repo-deadman-script> \
|
||||
bash deploy/trialrun/install-offhost-backup-receiver.sh install
|
||||
|
||||
# 生产：先只安装 disabled units；路径、项目、构建和 SSH 身份必须绑定活动实例。
|
||||
<explicit-project-data-build-ssh-and-webhook-file-env> \
|
||||
bash deploy/trialrun/install-automatic-backup-timer.sh install
|
||||
|
||||
# 首批真实备份、ARM 哈希和脱敏告警均经单独确认后，才允许启用三组 timer。
|
||||
<same-arm-explicit-env> bash deploy/trialrun/install-offhost-backup-receiver.sh enable
|
||||
<same-explicit-env> bash deploy/trialrun/install-automatic-backup-timer.sh enable
|
||||
```
|
||||
|
||||
`BACKUP_RUN_AT` 使用生产主机本地时区，ARM 归档按 UTC 日分桶。调整执行时刻的当天可能出现预期空窗；改动后必须人工确认次日 UTC 分桶、下一次 daily 运行和 26 小时 dead-man 窗口。真实告警验收必须真正停止一次源端 timer 或故意让一次传输失败，再确认 ARM 首次告警、重复降噪和恢复通知；只测试机器人能发消息不算通过。
|
||||
|
||||
卸载只撤销 timer/receiver 授权，保留源端批次、状态、ARM archive 和凭证文件；专用账号/密钥的最终销毁另走获批清理。
|
||||
|
||||
### 1.5.2 本地隔离恢复演练
|
||||
|
||||
开发机具备 Docker、PostgreSQL 客户端，以及 Node.js 或 Python 3 时，使用合成数据运行：
|
||||
@@ -238,7 +301,7 @@ RESTORE_SCOPE=shadow \
|
||||
|
||||
`restore-db.sh` 和 `restore-storage.sh` 默认只接受 `shadow`。影子库名必须包含独立的 `shadow`、`drill` 或 `restore_test` 标记；数据库确认不匹配、库名不安全、附件目录非空、目标为默认试运行附件目录或归档路径危险时均在写入前退出。
|
||||
|
||||
真实格式影子恢复的验收不仅是 `/ready`：恢复后的 `_prisma_migrations` 成功项必须等于冻结基线（当前预期 58），预先列出的关键业务表计数必须逐表一致，附件解包后的普通文件数必须与活动源目录一致。任一项不一致即判定批次不可作为切换回滚点。
|
||||
真实格式影子恢复的验收不仅是 `/ready`：恢复后的 `_prisma_migrations` 成功项必须等于执行前现场冻结基线（当前生产证据为 61 项；本地 62 项仅表示下一次部署预计新增 1 项），预先列出的关键业务表计数必须逐表一致，附件解包后的普通文件数必须与活动源目录一致。任一项不一致即判定批次不可作为切换回滚点。
|
||||
|
||||
### 1.5.3 生产切换前冻结记录
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#!/usr/bin/env node
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const childProcess = require('node:child_process');
|
||||
|
||||
const rootDir = path.resolve(__dirname, '..', '..');
|
||||
const configPath = path.join(rootDir, 'server/src/config.js');
|
||||
@@ -19,6 +20,18 @@ const timerScriptPath = path.resolve(
|
||||
process.env.ENV_CONTRACT_TIMER_SCRIPT_PATH
|
||||
|| path.join(rootDir, 'deploy/trialrun/install-warch3-ops-timer.sh'),
|
||||
);
|
||||
const automaticBackupTimerScriptPath = path.join(
|
||||
rootDir,
|
||||
'deploy/trialrun/install-automatic-backup-timer.sh',
|
||||
);
|
||||
const offhostReceiverInstallerPath = path.join(
|
||||
rootDir,
|
||||
'deploy/trialrun/install-offhost-backup-receiver.sh',
|
||||
);
|
||||
const automaticBackupRunnerPath = path.join(rootDir, 'scripts/run-automatic-backup.sh');
|
||||
const offhostReceiverPath = path.join(rootDir, 'scripts/receive-offhost-backup.sh');
|
||||
const groupAlertHelperPath = path.join(rootDir, 'scripts/send-wecom-group-alert.py');
|
||||
const offhostDeadmanPath = path.join(rootDir, 'scripts/check-offhost-backup-deadman.py');
|
||||
const nginxPaths = [
|
||||
path.join(rootDir, 'deploy/trialrun/nginx-hyb-oa.conf'),
|
||||
path.join(rootDir, 'deploy/trialrun/nginx.conf.example'),
|
||||
@@ -143,6 +156,12 @@ const runbookContent = readFile(runbookPath);
|
||||
const checklistContent = readFile(checklistPath);
|
||||
const deployScriptContent = readFile(deployScriptPath);
|
||||
const timerScriptContent = readFile(timerScriptPath);
|
||||
const automaticBackupTimerScriptContent = readFile(automaticBackupTimerScriptPath);
|
||||
const offhostReceiverInstallerContent = readFile(offhostReceiverInstallerPath);
|
||||
const automaticBackupRunnerContent = readFile(automaticBackupRunnerPath);
|
||||
const offhostReceiverContent = readFile(offhostReceiverPath);
|
||||
const groupAlertHelperContent = readFile(groupAlertHelperPath);
|
||||
const offhostDeadmanContent = readFile(offhostDeadmanPath);
|
||||
const nginxContents = nginxPaths.map((filePath) => ({
|
||||
filePath,
|
||||
content: readFile(filePath),
|
||||
@@ -171,6 +190,25 @@ for (const key of ['APP_HOST_PORT', 'POSTGRES_DATA_DIR', 'STORAGE_HOST_DIR']) {
|
||||
if (envExampleKeys.has('COMPOSE_PROJECT_NAME')) {
|
||||
errors.push('.env.example 不应持久化 COMPOSE_PROJECT_NAME，调用方必须通过 -p 显式选择项目');
|
||||
}
|
||||
for (const key of ['ALERT_WEBHOOK_FILE', 'WECOM_WEBHOOK_URL', 'WECOM_WEBHOOK_KEY']) {
|
||||
if (envExampleKeys.has(key)) {
|
||||
errors.push(`.env.example 不应持久化企微群机器人配置 ${key}`);
|
||||
}
|
||||
}
|
||||
const trackedFiles = childProcess.execFileSync('git', ['ls-files', '--cached', '--others', '--exclude-standard', '-z'], {
|
||||
cwd: rootDir,
|
||||
encoding: 'utf8',
|
||||
}).split('\0').filter(Boolean);
|
||||
const literalWebhookPattern = /https:\/\/qyapi\.weixin\.qq\.com\/cgi-bin\/webhook\/send\?key=/;
|
||||
for (const relativePath of trackedFiles) {
|
||||
const filePath = path.join(rootDir, relativePath);
|
||||
if (!fs.statSync(filePath).isFile()) continue;
|
||||
const content = fs.readFileSync(filePath);
|
||||
if (content.includes(0)) continue;
|
||||
if (literalWebhookPattern.test(content.toString('utf8'))) {
|
||||
errors.push(`${relativePath} 不得持久化完整企微群机器人 Webhook URL`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const serviceName of requiredServices) {
|
||||
const block = extractServiceBlock(composeContent, serviceName);
|
||||
@@ -241,6 +279,152 @@ for (const [name, content] of [
|
||||
if (!content.includes('-p "$COMPOSE_PROJECT_NAME"')) {
|
||||
errors.push(`${name} 必须向 Compose 显式传入独立 project name`);
|
||||
}
|
||||
if (!content.includes('"$value" == *"//"*')
|
||||
|| !content.includes('"/$value/" == *"/../"*')
|
||||
|| !content.includes('"/$value/" == *"/./"*')) {
|
||||
errors.push(`${name} 缺少绝对数据路径的空段与点路径段门禁`);
|
||||
}
|
||||
if (content.includes('[[:space:]]*)($|/)')) {
|
||||
errors.push(`${name} 不得以可匹配空字符串的路径段正则拒绝全部绝对路径`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'POSTGRES_DATA_DIR="${POSTGRES_DATA_DIR:-}"',
|
||||
'STORAGE_HOST_DIR="${STORAGE_HOST_DIR:-}"',
|
||||
'BUILD_SHA="${BUILD_SHA:-}"',
|
||||
'VITE_BUILD_SHA="${VITE_BUILD_SHA:-}"',
|
||||
'validate_data_path POSTGRES_DATA_DIR "$POSTGRES_DATA_DIR"',
|
||||
'validate_data_path STORAGE_HOST_DIR "$STORAGE_HOST_DIR"',
|
||||
'"$POSTGRES_DATA_DIR" == "$STORAGE_HOST_DIR"/*',
|
||||
'"$STORAGE_HOST_DIR" == "$POSTGRES_DATA_DIR"/*',
|
||||
'"$POSTGRES_DATA_DIR/PG_VERSION"',
|
||||
'"$BUILD_SHA" =~ ^[0-9a-fA-F]{40,64}$',
|
||||
'"$VITE_BUILD_SHA" =~ ^[0-9a-f]{12}$',
|
||||
'"${BUILD_SHA:0:12}" != "$VITE_BUILD_SHA"',
|
||||
'Environment=POSTGRES_DATA_DIR=${POSTGRES_DATA_DIR}',
|
||||
'Environment=STORAGE_HOST_DIR=${STORAGE_HOST_DIR}',
|
||||
'Environment=BUILD_SHA=${BUILD_SHA}',
|
||||
'Environment=VITE_BUILD_SHA=${VITE_BUILD_SHA}',
|
||||
]) {
|
||||
if (!timerScriptContent.includes(requiredFragment)) {
|
||||
errors.push(`install-warch3-ops-timer.sh 缺少活动部署环境绑定: ${requiredFragment}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'MODE="${1:-install}"',
|
||||
'write_service daily run',
|
||||
'write_service watchdog check-stale',
|
||||
'write_service drill-reminder remind-drill',
|
||||
'write_timer daily',
|
||||
'write_timer watchdog',
|
||||
'write_timer drill-reminder',
|
||||
'Environment=COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME}',
|
||||
'Environment=POSTGRES_DATA_DIR=${POSTGRES_DATA_DIR}',
|
||||
'Environment=STORAGE_HOST_DIR=${STORAGE_HOST_DIR}',
|
||||
'Environment=BUILD_SHA=${BUILD_SHA}',
|
||||
'Environment=VITE_BUILD_SHA=${VITE_BUILD_SHA}',
|
||||
'Environment=AUTOMATIC_BACKUP_DIR=${AUTOMATIC_BACKUP_DIR}',
|
||||
'Environment=AUTOMATIC_BACKUP_STATE_DIR=${AUTOMATIC_BACKUP_STATE_DIR}',
|
||||
'Environment=OFFHOST_TARGET=${OFFHOST_TARGET}',
|
||||
'Environment=OFFHOST_SSH_KEY=${OFFHOST_SSH_KEY}',
|
||||
'Environment=OFFHOST_KNOWN_HOSTS=${OFFHOST_KNOWN_HOSTS}',
|
||||
'Environment=ALERT_WEBHOOK_FILE=${ALERT_WEBHOOK_FILE}',
|
||||
'Environment=ALERT_HELPER=${ALERT_HELPER}',
|
||||
'install)\n install_units',
|
||||
'enable)\n enable_units',
|
||||
]) {
|
||||
if (!automaticBackupTimerScriptContent.includes(requiredFragment)) {
|
||||
errors.push(`install-automatic-backup-timer.sh 缺少自动备份安装合同: ${requiredFragment}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'COMPOSE_CMD=(docker compose)',
|
||||
'-p "$COMPOSE_PROJECT_NAME"',
|
||||
'flock -n 9',
|
||||
'"$actual_storage" == "$STORAGE_HOST_DIR"',
|
||||
'"$actual_pgdata" == "$POSTGRES_DATA_DIR"',
|
||||
'--partial --chmod=Du=rwx,Dgo=,Fu=rw,Fgo=',
|
||||
'StrictHostKeyChecking=yes',
|
||||
'hyb-backup-finalize $run_id $manifest_sha',
|
||||
'write_success_state "$run_id" "$manifest_sha"',
|
||||
'prune_local_backups "$batch_name"',
|
||||
'python3 "$ALERT_HELPER"',
|
||||
'--webhook-file "$ALERT_WEBHOOK_FILE"',
|
||||
'capture_command offhost_transfer rsync',
|
||||
'last-error.log',
|
||||
"docker inspect -f '{{json .Mounts}}'",
|
||||
'if len(matches) != 1 or not matches[0]:',
|
||||
]) {
|
||||
if (!automaticBackupRunnerContent.includes(requiredFragment)) {
|
||||
errors.push(`run-automatic-backup.sh 缺少运行时安全合同: ${requiredFragment}`);
|
||||
}
|
||||
}
|
||||
if (automaticBackupRunnerContent.includes('println .Destination "\\t" .Source')) {
|
||||
errors.push('run-automatic-backup.sh 不得用人类可读 Go-template 分隔符解析真实 Docker mount');
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'"$original_command" == rsync\\ --server\\ *',
|
||||
'--sender([[:space:]]|$)',
|
||||
'exec "$RRSYNC_BIN" -wo "$BACKUP_INBOX_DIR"',
|
||||
'hyb-backup-finalize',
|
||||
'batch must contain exactly three regular files',
|
||||
'manifest artifact checksum mismatch',
|
||||
'mv -- "$partial_dir" "$archive_dir"',
|
||||
'replacement_rollback=',
|
||||
'publish_weekly_checkpoint',
|
||||
'prune_named_directories "$BACKUP_ARCHIVE_DIR"',
|
||||
'prune_named_directories "$BACKUP_WEEKLY_DIR"',
|
||||
]) {
|
||||
if (!offhostReceiverContent.includes(requiredFragment)) {
|
||||
errors.push(`receive-offhost-backup.sh 缺少受限接收合同: ${requiredFragment}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'MODE="${1:-install}"',
|
||||
'passwd -l "$BACKUP_USER"',
|
||||
'command="${RECEIVER_TARGET}",no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-pty,no-user-rc',
|
||||
'BACKUP_INBOX_DIR=${inbox_dir}',
|
||||
'BACKUP_ARCHIVE_DIR=${archive_dir}',
|
||||
'BACKUP_WEEKLY_DIR=${weekly_dir}',
|
||||
'DEADMAN_MAX_AGE_SECONDS="${DEADMAN_MAX_AGE_SECONDS:-93600}"',
|
||||
'ExecStart=/usr/bin/python3 ${DEADMAN_TARGET}',
|
||||
'systemctl enable --now "${DEADMAN_SERVICE}.timer"',
|
||||
'archives retained',
|
||||
'backup root contains an unmanaged top-level entry',
|
||||
'backup root must not be inside a system-managed directory',
|
||||
]) {
|
||||
if (!offhostReceiverInstallerContent.includes(requiredFragment)) {
|
||||
errors.push(`install-offhost-backup-receiver.sh 缺少受限身份安装合同: ${requiredFragment}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'stat.S_IMODE(info.st_mode) != 0o600',
|
||||
'parsed.hostname != "qyapi.weixin.qq.com"',
|
||||
'HYB_WECOM_ALERT_TEST_MODE',
|
||||
'should_send(previous, status, category)',
|
||||
'不要在群内粘贴原始日志',
|
||||
]) {
|
||||
if (!groupAlertHelperContent.includes(requiredFragment)) {
|
||||
errors.push(`send-wecom-group-alert.py 缺少独立脱敏告警合同: ${requiredFragment}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
'default=93600',
|
||||
'archive_manifest_invalid',
|
||||
'archive_time_in_future',
|
||||
'FAIL_OFFHOST_DEADMAN',
|
||||
'deadman-alert-state.json',
|
||||
]) {
|
||||
if (!offhostDeadmanContent.includes(requiredFragment)) {
|
||||
errors.push(`check-offhost-backup-deadman.py 缺少 ARM 独立看门狗合同: ${requiredFragment}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const requiredFragment of [
|
||||
|
||||
+2
-1
@@ -40,9 +40,10 @@
|
||||
| 企业微信回调 IP | [获取企业微信回调 IP 段](https://developer.work.weixin.qq.com/document/path/92521) | `/cgi-bin/getcallbackip` | `GET` | `WECOM_AGENT_SECRET` | `CAN_REUSE_NOW` | 用于回调入口放行和安全审计，不进入业务数据。 |
|
||||
| 网页授权链接 | [构造网页授权链接](https://developer.work.weixin.qq.com/document/path/91022) | `connect/oauth2/authorize` | 浏览器跳转 | `WECOM_AGENT_SECRET` 所属应用 | `CAN_REUSE_NOW` | 企业微信工作台入口和 deeplink 回 OA 使用；只改变登录体验，不改变 OA 权限。 |
|
||||
| 访问用户身份 | [获取访问用户身份](https://developer.work.weixin.qq.com/document/path/91023) | `/cgi-bin/auth/getuserinfo` | `GET` | `WECOM_AGENT_SECRET` | `CAN_REUSE_NOW` | 获取当前企业微信 userid 后匹配 OA 绑定；不能仅靠手机号/邮箱授权登录。 |
|
||||
| 成员本人敏感资料授权 | [网页授权链接](https://developer.work.weixin.qq.com/document/path/91022)、[访问用户身份](https://developer.work.weixin.qq.com/document/path/91023)、[访问用户敏感信息](https://developer.work.weixin.qq.com/document/path/95833) | `snsapi_privateinfo` + `/cgi-bin/auth/getuserinfo` + `/cgi-bin/auth/getuserdetail` | 浏览器跳转 + `GET/POST` | `WECOM_AGENT_SECRET` 所属应用 | `LOCAL_IMPLEMENTED_EXTERNAL_CONFIG_PENDING` | 普通登录继续用 `snsapi_base`；仅默认关闭的一次性开通入口请求私密授权。成员同意后，服务端用短时 `user_ticket` 立即换取手机及个人邮箱或企业邮箱；票据、code 和原始响应不落库、不写 Cookie、日志或审计。管理员仍需在应用详情选择相应敏感字段，成员需本人确认。 |
|
||||
| 成员本人敏感资料授权 | [网页授权链接](https://developer.work.weixin.qq.com/document/path/91022)、[访问用户身份](https://developer.work.weixin.qq.com/document/path/91023)、[访问用户敏感信息](https://developer.work.weixin.qq.com/document/path/95833) | `snsapi_privateinfo` + `/cgi-bin/auth/getuserinfo` + `/cgi-bin/auth/getuserdetail` | 浏览器跳转 + `GET/POST` | `WECOM_AGENT_SECRET` 所属应用 | `LOCAL_IMPLEMENTED_EXTERNAL_CONFIG_PENDING` | 普通登录继续用 `snsapi_base`；仅默认关闭的一次性开通入口请求私密授权。成员同意后，服务端用短时 `user_ticket` 立即换取已授权字段；个人邮箱或企业邮箱仍是建档必填，手机号可暂缺，返回时才规范化并检查唯一。票据、code 和原始响应不落库、不写 Cookie、日志或审计。管理员仍需在应用详情选择相应敏感字段，成员需本人确认。 |
|
||||
| 应用消息 | [发送应用消息](https://developer.work.weixin.qq.com/document/path/90236) | `/cgi-bin/message/send` | `POST` | `WECOM_AGENT_SECRET` | `CAN_REUSE_NOW` | OA 主办审批只发脱敏标题、类型、编号、状态和 OA deeplink；不发送高敏附件。 |
|
||||
| 通讯录概述 | [通讯录管理](https://developer.work.weixin.qq.com/document/path/90193) | 通讯录成员、部门、回调和导入导出 | 混合 | `WECOM_CONTACT_SECRET` | `CAN_REUSE_NOW_PARTIAL` | 通讯录是单独 Secret 域，和工作台应用 Secret 分开管理。 |
|
||||
| 管理后台通讯录导出 | 企业微信管理后台导出 XLSX | `npm run import:wecom-directory-export -- --input=- --dry-run` | 本地 stdin | 无 Secret | `LOCAL_IMPLEMENTED_NOT_DEPLOYED` | 作为批量 API 缺少敏感字段时的一次性管理员输入；只接受 stdin、默认 dry-run，真实 apply 需参数和环境变量双门及另行批准。企微账号和企业邮箱必填且唯一，手机号可暂缺并保存为 `NULL`；空来源不清除 OA 既有手机号，报告只输出计数和类别，原文件不入 Git。 |
|
||||
| 成员 ID 清单 | [获取成员 ID 列表](https://developer.work.weixin.qq.com/document/path/96067) | `/cgi-bin/user/list_id` | `POST` | `WECOM_CONTACT_SECRET` | `CAN_REUSE_NOW` | WOUA/WHMS 全量候选必须走该接口，请求体只使用 `cursor`/`limit`，返回用户-部门关系后本地按 userid 去重；不再用旧 `/cgi-bin/user/list` 做全量读取。 |
|
||||
| 读取成员 | [读取成员](https://developer.work.weixin.qq.com/document/path/90196) | `/cgi-bin/user/get` | `GET` | `WECOM_AGENT_SECRET` | `OPTIONAL_ENRICHMENT` | 用自建应用 token 补齐 userid 对应的姓名、状态和部门等可见资料。2022-06-20 后新建应用的手机、邮箱和企业邮箱不能以批量接口缺失推断为企微不支持；这些字段按成员本人敏感资料授权链路获取。 |
|
||||
| 获取部门成员 | [获取部门成员](https://developer.work.weixin.qq.com/document/path/90200) | `/cgi-bin/user/simplelist` | `GET` | `WECOM_CONTACT_SECRET` | `LEGACY_NOT_DEFAULT` | 仅作历史参考，当前全量读取不依赖它。 |
|
||||
|
||||
+11
@@ -356,6 +356,17 @@ WAPC2 请假/补卡只读 POC 操作口径：
|
||||
6. 管理员查看审计日志中是否有异常导出、权限拒绝或付款动作。
|
||||
7. 记录当天新增问题、影响等级、处理人、结论和是否需要暂停。
|
||||
|
||||
自动异机备份运维口径：
|
||||
|
||||
1. 自动备份以 OA 仓库 runbook 首部“紧急恢复卡”为唯一短入口；公司当前只有一名系统管理员，由该管理员维护恢复卡，不另设知识库副本或虚构替补操作人。
|
||||
2. 每日 02:30 生成数据库、附件、manifest 同批恢复点并推送 ARM；07:10 检查最近 24 小时异机成功状态，ARM 另以 26 小时 dead-man 独立检查归档。生产保留 7 个自动日批次，ARM 保留 7 个日批次和 4 个周检查点。
|
||||
3. ARM 专用身份只能写 inbox 和请求 finalize，不能登录 shell、读取或删除 archive。任何人不得为了省事改用普通 ARM 登录账号或关闭 known-hosts 固定校验。
|
||||
4. 自动备份使用独立企微运维群机器人，不依赖 OA server 或个人员工号。Webhook 只由用户写入两台主机的 root-only 文件，不进仓库、普通 `.env`、日志或聊天。企微只收脱敏异常、finding 和恢复通知；同一失败降噪。季度提醒只要求人工在 ARM 隔离目标恢复，不会自动恢复生产。
|
||||
5. 群机器人、ARM dead-man、更新后的两端脚本、首次有效真实批次、全部 timer enable 和真实源端停止告警均已按阶段授权完成并回填当前 Trellis task；后续安装或配置变更仍为 **EXTERNAL_GATED**。真实验收必须停止一次源端 timer 或故意让一次传输失败，不能只测试机器人发送。
|
||||
6. 每季度由唯一系统管理员按恢复卡在 ARM 隔离目标执行一次影子恢复，核对归档类别、三条恢复入口、`/ready` 和关键计数；未实际执行时，本季度恢复门禁不通过。
|
||||
7. 卸载 timer 或 receiver 时保留历史 archive 和源端完整批次；账号、密钥、备份数据的删除必须另行批准。
|
||||
8. 改动每日备份时刻后，人工确认次日 UTC 分桶和 26 小时窗口，避免把调整日的预期空窗误当成长期故障。
|
||||
|
||||
WROAD2/WARCH3 企微受控上线每日巡检：
|
||||
|
||||
1. 试用范围固定为测试账号 `SMOKE002 -> SMOKE101`，管理员观察和告警收件人为 `HYD000006`；`HYD000006` 是 `wangshunli` 的员工号，`wangshunli` 只作为姓名/账号别名记录。试用周期 3 天；WARCH3 可通过 `hyb-oa-warch3-ops.timer` 每天 07:30 自动触发。
|
||||
|
||||
+1
-1
@@ -71,7 +71,7 @@ if [[ -z "$POSTGRES_DB" || -z "$POSTGRES_USER" || -z "$POSTGRES_PASSWORD" ]]; th
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \
|
||||
if ! PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD "$DB_CONTAINER" \
|
||||
pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" --format=c --no-owner --no-acl \
|
||||
>"$OUT_FILE"; then
|
||||
rm -f -- "$OUT_FILE"
|
||||
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate the newest ARM archive and drive an independent dead-man alert."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
BATCH_RE = re.compile(r"^hyb-oa-backup-(auto-(\d{8}T\d{6}Z))$")
|
||||
|
||||
|
||||
def fail(message: str) -> "NoReturn":
|
||||
print(f"[offhost-deadman][ERROR] {message}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def validate_latest(archive_root: Path, max_age: int) -> tuple[str, str]:
|
||||
if not archive_root.is_dir() or archive_root.is_symlink():
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_root_invalid"
|
||||
candidates = sorted(
|
||||
(entry for entry in archive_root.iterdir() if entry.is_dir() and not entry.is_symlink() and BATCH_RE.fullmatch(entry.name)),
|
||||
key=lambda entry: entry.name,
|
||||
)
|
||||
if not candidates:
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_missing"
|
||||
batch = candidates[-1]
|
||||
match = BATCH_RE.fullmatch(batch.name)
|
||||
assert match is not None
|
||||
run_id, stamp = match.groups()
|
||||
expected_names = {"database.dump", "storage.tar.gz", "manifest.json"}
|
||||
entries = list(batch.iterdir())
|
||||
if {entry.name for entry in entries} != expected_names or any(
|
||||
entry.is_symlink() or not stat.S_ISREG(entry.stat().st_mode) for entry in entries
|
||||
):
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_files_invalid"
|
||||
try:
|
||||
manifest = json.loads((batch / "manifest.json").read_text(encoding="utf-8"))
|
||||
if manifest.get("schemaVersion") != 1 or manifest.get("status") != "complete" or manifest.get("runId") != run_id:
|
||||
raise ValueError
|
||||
artifacts = manifest.get("artifacts") or {}
|
||||
for key, name in (("database", "database.dump"), ("storage", "storage.tar.gz")):
|
||||
item = artifacts.get(key) or {}
|
||||
target = batch / name
|
||||
if item.get("file") != name or item.get("bytes") != target.stat().st_size or item.get("sha256") != sha256(target):
|
||||
raise ValueError
|
||||
completed = datetime.strptime(stamp, "%Y%m%dT%H%M%SZ").replace(tzinfo=timezone.utc)
|
||||
except (OSError, ValueError, TypeError, json.JSONDecodeError):
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_manifest_invalid"
|
||||
age = (datetime.now(timezone.utc) - completed).total_seconds()
|
||||
if age < 0:
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_time_in_future"
|
||||
if age > max_age:
|
||||
return "FAIL_OFFHOST_DEADMAN", "archive_stale"
|
||||
return "PASS_OFFHOST_DEADMAN_FRESH", "fresh_offhost_archive"
|
||||
|
||||
|
||||
def write_summary(path: Path, status: str, category: str) -> None:
|
||||
finding = status.startswith("FAIL_")
|
||||
payload = {
|
||||
"issueId": "AUTO_BACKUP",
|
||||
"status": status,
|
||||
"conclusion": status,
|
||||
"findingCount": 1 if finding else 0,
|
||||
"findings": ([{"source": "offhost_deadman", "status": status, "severity": "P0", "detail": category}] if finding else []),
|
||||
"privacy": "status/count/category only; no credentials, host addresses, absolute paths, file names, or payloads",
|
||||
}
|
||||
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
fd, temporary = tempfile.mkstemp(prefix=".deadman-summary.", dir=path.parent)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, ensure_ascii=True, indent=2)
|
||||
handle.write("\n")
|
||||
os.chmod(temporary, 0o600)
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--archive-root", required=True, type=Path)
|
||||
parser.add_argument("--state-dir", required=True, type=Path)
|
||||
parser.add_argument("--webhook-file", required=True, type=Path)
|
||||
parser.add_argument("--alert-helper", required=True, type=Path)
|
||||
parser.add_argument("--max-age-seconds", type=int, default=93600)
|
||||
args = parser.parse_args()
|
||||
if args.max_age_seconds <= 0:
|
||||
fail("max age must be positive")
|
||||
try:
|
||||
args.state_dir.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
if args.state_dir.is_symlink() or not args.state_dir.is_dir():
|
||||
fail("state directory is invalid")
|
||||
os.chmod(args.state_dir, 0o700)
|
||||
status, category = validate_latest(args.archive_root, args.max_age_seconds)
|
||||
summary = args.state_dir / "last-deadman.json"
|
||||
write_summary(summary, status, category)
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(args.alert_helper), "--webhook-file", str(args.webhook_file), "--summary-file", str(summary), "--state-file", str(args.state_dir / "deadman-alert-state.json")],
|
||||
check=False,
|
||||
)
|
||||
except OSError:
|
||||
fail("dead-man local file operation failed")
|
||||
if result.returncode != 0:
|
||||
fail("alert evaluation failed")
|
||||
if status.startswith("FAIL_"):
|
||||
fail(category)
|
||||
print("[offhost-deadman][PASS] newest archive is fresh and valid")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+726
@@ -0,0 +1,726 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
TEST_DIR="$(mktemp -d)"
|
||||
PASS_COUNT=0
|
||||
|
||||
cleanup() {
|
||||
if [[ -n "${TEST_DIR:-}" && -d "$TEST_DIR" && "$TEST_DIR" == /tmp/* ]]; then
|
||||
rm -rf -- "$TEST_DIR"
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
pass() {
|
||||
PASS_COUNT=$((PASS_COUNT + 1))
|
||||
printf '[PASS] %s\n' "$1"
|
||||
}
|
||||
|
||||
expect_failure() {
|
||||
local label="$1"
|
||||
shift
|
||||
if "$@" >"$TEST_DIR/last-command.out" 2>&1; then
|
||||
printf '[FAIL] expected rejection: %s\n' "$label" >&2
|
||||
exit 1
|
||||
fi
|
||||
pass "$label"
|
||||
}
|
||||
|
||||
make_layout() {
|
||||
local root="$1"
|
||||
mkdir -p "$root/inbox" "$root/archive" "$root/weekly"
|
||||
}
|
||||
|
||||
make_batch() {
|
||||
local target="$1"
|
||||
local run_id="$2"
|
||||
mkdir -p "$target"
|
||||
printf 'synthetic database payload\n' >"$target/database.dump"
|
||||
printf 'synthetic storage payload\n' >"$target/storage.tar.gz"
|
||||
python3 - "$target" "$run_id" <<'PYTHON'
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
target, run_id = sys.argv[1:]
|
||||
def artifact(file_name):
|
||||
file_path = os.path.join(target, file_name)
|
||||
with open(file_path, "rb") as handle:
|
||||
content = handle.read()
|
||||
return {"file": file_name, "bytes": len(content), "sha256": hashlib.sha256(content).hexdigest()}
|
||||
|
||||
manifest = {
|
||||
"schemaVersion": 1,
|
||||
"runId": run_id,
|
||||
"createdAt": "2026-01-01T00:00:00Z",
|
||||
"buildSha": "a" * 40,
|
||||
"sourceImageDigest": None,
|
||||
"status": "complete",
|
||||
"artifacts": {
|
||||
"database": artifact("database.dump"),
|
||||
"storage": artifact("storage.tar.gz"),
|
||||
},
|
||||
}
|
||||
with open(os.path.join(target, "manifest.json"), "x", encoding="utf-8") as handle:
|
||||
json.dump(manifest, handle, indent=2)
|
||||
handle.write("\n")
|
||||
PYTHON
|
||||
}
|
||||
|
||||
receiver() {
|
||||
local layout="$1"
|
||||
shift
|
||||
env RECEIVER_TEST_MODE=1 RECEIVER_CONFIG_FILE="$TEST_DIR/no-receiver-config" \
|
||||
BACKUP_INBOX_DIR="$layout/inbox" BACKUP_ARCHIVE_DIR="$layout/archive" \
|
||||
BACKUP_WEEKLY_DIR="$layout/weekly" DAILY_KEEP=7 WEEKLY_KEEP=4 \
|
||||
"$ROOT_DIR/scripts/receive-offhost-backup.sh" "$@"
|
||||
}
|
||||
|
||||
CURRENT_RUN_ID="auto-$(date -u +%Y%m%d)T000000Z"
|
||||
CURRENT_BATCH="hyb-oa-backup-$CURRENT_RUN_ID"
|
||||
|
||||
VALID_LAYOUT="$TEST_DIR/receiver-valid"
|
||||
make_layout "$VALID_LAYOUT"
|
||||
make_batch "$VALID_LAYOUT/inbox/${CURRENT_BATCH}.partial" "$CURRENT_RUN_ID"
|
||||
VALID_MANIFEST_SHA="$(sha256sum "$VALID_LAYOUT/inbox/${CURRENT_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
receiver "$VALID_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$VALID_MANIFEST_SHA" >/dev/null
|
||||
[[ -d "$VALID_LAYOUT/archive/$CURRENT_BATCH" && ! -e "$VALID_LAYOUT/inbox/${CURRENT_BATCH}.partial" ]]
|
||||
pass "receiver finalizes one complete synthetic batch"
|
||||
|
||||
receiver "$VALID_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$VALID_MANIFEST_SHA" >/dev/null
|
||||
pass "receiver treats an identical repeated finalize as idempotent"
|
||||
|
||||
SECOND_RUN_ID="auto-$(date -u +%Y%m%d)T000001Z"
|
||||
SECOND_BATCH="hyb-oa-backup-$SECOND_RUN_ID"
|
||||
make_batch "$VALID_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID"
|
||||
SECOND_MANIFEST_SHA="$(sha256sum "$VALID_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
receiver "$VALID_LAYOUT" finalize-test "$SECOND_RUN_ID" "$SECOND_MANIFEST_SHA" >/dev/null
|
||||
[[ ! -e "$VALID_LAYOUT/archive/$CURRENT_BATCH" && -d "$VALID_LAYOUT/archive/$SECOND_BATCH" ]]
|
||||
WEEKLY_BATCH="$(find "$VALID_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -print -quit)"
|
||||
[[ -n "$WEEKLY_BATCH" ]]
|
||||
[[ "$(stat -c %i "$WEEKLY_BATCH/database.dump")" == "$(stat -c %i "$VALID_LAYOUT/archive/$SECOND_BATCH/database.dump")" ]]
|
||||
pass "receiver atomically replaces a different batch for the same UTC day and rebuilds its weekly checkpoint"
|
||||
|
||||
THIRD_RUN_ID="auto-$(date -u +%Y%m%d)T000002Z"
|
||||
THIRD_BATCH="hyb-oa-backup-$THIRD_RUN_ID"
|
||||
make_batch "$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID"
|
||||
printf 'corrupt candidate\n' >>"$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial/database.dump"
|
||||
THIRD_MANIFEST_SHA="$(sha256sum "$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver validates a same-day replacement before touching the current archive" \
|
||||
receiver "$VALID_LAYOUT" finalize-test "$THIRD_RUN_ID" "$THIRD_MANIFEST_SHA"
|
||||
[[ -d "$VALID_LAYOUT/archive/$SECOND_BATCH" && ! -e "$VALID_LAYOUT/archive/$THIRD_BATCH" ]]
|
||||
pass "receiver preserves the current daily archive when a replacement candidate is corrupt"
|
||||
|
||||
ROLLBACK_LAYOUT="$TEST_DIR/receiver-replacement-rollback"
|
||||
make_layout "$ROLLBACK_LAYOUT"
|
||||
make_batch "$ROLLBACK_LAYOUT/inbox/${CURRENT_BATCH}.partial" "$CURRENT_RUN_ID"
|
||||
ROLLBACK_FIRST_SHA="$(sha256sum "$ROLLBACK_LAYOUT/inbox/${CURRENT_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
receiver "$ROLLBACK_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$ROLLBACK_FIRST_SHA" >/dev/null
|
||||
ROLLBACK_WEEKLY="$(find "$ROLLBACK_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -print -quit)"
|
||||
find "$ROLLBACK_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -exec rm -rf -- {} +
|
||||
printf 'invalid weekly target\n' >"$ROLLBACK_WEEKLY"
|
||||
make_batch "$ROLLBACK_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID"
|
||||
ROLLBACK_SECOND_SHA="$(sha256sum "$ROLLBACK_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver rolls back the daily archive when weekly checkpoint publication fails" \
|
||||
receiver "$ROLLBACK_LAYOUT" finalize-test "$SECOND_RUN_ID" "$ROLLBACK_SECOND_SHA"
|
||||
[[ -d "$ROLLBACK_LAYOUT/archive/$CURRENT_BATCH" && ! -e "$ROLLBACK_LAYOUT/archive/$SECOND_BATCH" ]]
|
||||
pass "receiver restores the prior daily point after a later replacement step fails"
|
||||
|
||||
CURRENT_WEEK="$(python3 - "$CURRENT_RUN_ID" <<'PYTHON'
|
||||
from datetime import datetime
|
||||
import sys
|
||||
day = datetime.strptime(sys.argv[1][5:13], "%Y%m%d").date().isocalendar()
|
||||
print(f"week-{day.year:04d}-W{day.week:02d}")
|
||||
PYTHON
|
||||
)"
|
||||
|
||||
make_corrupt_partial() {
|
||||
local layout="$1"
|
||||
local run_id="$2"
|
||||
local batch_name="hyb-oa-backup-$run_id"
|
||||
make_batch "$layout/inbox/${batch_name}.partial" "$run_id"
|
||||
printf 'corrupt candidate\n' >>"$layout/inbox/${batch_name}.partial/database.dump"
|
||||
sha256sum "$layout/inbox/${batch_name}.partial/manifest.json" | awk '{print $1}'
|
||||
}
|
||||
|
||||
DAILY_COMPLETE_LAYOUT="$TEST_DIR/receiver-daily-orphan-complete"
|
||||
make_layout "$DAILY_COMPLETE_LAYOUT"
|
||||
make_batch "$DAILY_COMPLETE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.101" "$CURRENT_RUN_ID"
|
||||
make_batch "$DAILY_COMPLETE_LAYOUT/archive/$SECOND_BATCH" "$SECOND_RUN_ID"
|
||||
DAILY_COMPLETE_SHA="$(make_corrupt_partial "$DAILY_COMPLETE_LAYOUT" "$THIRD_RUN_ID")"
|
||||
expect_failure "receiver cleans a valid daily rollback after the replacement was published" \
|
||||
receiver "$DAILY_COMPLETE_LAYOUT" finalize-test "$THIRD_RUN_ID" "$DAILY_COMPLETE_SHA"
|
||||
[[ ! -e "$DAILY_COMPLETE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.101" \
|
||||
&& -d "$DAILY_COMPLETE_LAYOUT/archive/$SECOND_BATCH" ]]
|
||||
[[ -d "$DAILY_COMPLETE_LAYOUT/weekly/$CURRENT_WEEK" ]]
|
||||
pass "receiver keeps the valid published daily point and rebuilds its weekly checkpoint"
|
||||
|
||||
DAILY_WEEKLY_FAIL_LAYOUT="$TEST_DIR/receiver-daily-orphan-weekly-failure"
|
||||
make_layout "$DAILY_WEEKLY_FAIL_LAYOUT"
|
||||
DAILY_WEEKLY_FAIL_ORPHAN="$DAILY_WEEKLY_FAIL_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.107"
|
||||
make_batch "$DAILY_WEEKLY_FAIL_ORPHAN" "$CURRENT_RUN_ID"
|
||||
make_batch "$DAILY_WEEKLY_FAIL_LAYOUT/archive/$SECOND_BATCH" "$SECOND_RUN_ID"
|
||||
printf 'invalid weekly target\n' >"$DAILY_WEEKLY_FAIL_LAYOUT/weekly/$CURRENT_WEEK"
|
||||
DAILY_WEEKLY_FAIL_SHA="$(make_corrupt_partial "$DAILY_WEEKLY_FAIL_LAYOUT" "$THIRD_RUN_ID")"
|
||||
expect_failure "receiver keeps a daily rollback until weekly reconciliation succeeds" \
|
||||
receiver "$DAILY_WEEKLY_FAIL_LAYOUT" finalize-test "$THIRD_RUN_ID" "$DAILY_WEEKLY_FAIL_SHA"
|
||||
[[ -d "$DAILY_WEEKLY_FAIL_ORPHAN" \
|
||||
&& -d "$DAILY_WEEKLY_FAIL_LAYOUT/archive/$SECOND_BATCH" \
|
||||
&& -f "$DAILY_WEEKLY_FAIL_LAYOUT/weekly/$CURRENT_WEEK" ]]
|
||||
pass "receiver preserves the daily rollback when weekly reconciliation fails"
|
||||
|
||||
DAILY_RESTORE_LAYOUT="$TEST_DIR/receiver-daily-orphan-restore"
|
||||
make_layout "$DAILY_RESTORE_LAYOUT"
|
||||
make_batch "$DAILY_RESTORE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.102" "$CURRENT_RUN_ID"
|
||||
DAILY_RESTORE_SHA="$(make_corrupt_partial "$DAILY_RESTORE_LAYOUT" "$SECOND_RUN_ID")"
|
||||
expect_failure "receiver restores a valid daily rollback when the published archive is absent" \
|
||||
receiver "$DAILY_RESTORE_LAYOUT" finalize-test "$SECOND_RUN_ID" "$DAILY_RESTORE_SHA"
|
||||
[[ -d "$DAILY_RESTORE_LAYOUT/archive/$CURRENT_BATCH" \
|
||||
&& ! -e "$DAILY_RESTORE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.102" ]]
|
||||
[[ -d "$DAILY_RESTORE_LAYOUT/weekly/$CURRENT_WEEK" ]]
|
||||
pass "receiver restores the last daily point before rejecting the next corrupt candidate"
|
||||
|
||||
DAILY_AMBIGUOUS_LAYOUT="$TEST_DIR/receiver-daily-orphan-ambiguous"
|
||||
make_layout "$DAILY_AMBIGUOUS_LAYOUT"
|
||||
make_batch "$DAILY_AMBIGUOUS_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.103" "$CURRENT_RUN_ID"
|
||||
make_batch "$DAILY_AMBIGUOUS_LAYOUT/archive/$SECOND_BATCH" "$SECOND_RUN_ID"
|
||||
printf 'corruption\n' >>"$DAILY_AMBIGUOUS_LAYOUT/archive/$SECOND_BATCH/database.dump"
|
||||
make_batch "$DAILY_AMBIGUOUS_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID"
|
||||
DAILY_AMBIGUOUS_SHA="$(sha256sum "$DAILY_AMBIGUOUS_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver fails closed when a daily rollback and invalid published archive coexist" \
|
||||
receiver "$DAILY_AMBIGUOUS_LAYOUT" finalize-test "$THIRD_RUN_ID" "$DAILY_AMBIGUOUS_SHA"
|
||||
[[ -d "$DAILY_AMBIGUOUS_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.103" \
|
||||
&& -d "$DAILY_AMBIGUOUS_LAYOUT/archive/$SECOND_BATCH" ]]
|
||||
pass "receiver preserves both daily objects when replacement state is ambiguous"
|
||||
|
||||
DAILY_DUPLICATE_LAYOUT="$TEST_DIR/receiver-daily-orphan-duplicate"
|
||||
make_layout "$DAILY_DUPLICATE_LAYOUT"
|
||||
make_batch "$DAILY_DUPLICATE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.104" "$CURRENT_RUN_ID"
|
||||
make_batch "$DAILY_DUPLICATE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.105" "$CURRENT_RUN_ID"
|
||||
make_batch "$DAILY_DUPLICATE_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID"
|
||||
DAILY_DUPLICATE_SHA="$(sha256sum "$DAILY_DUPLICATE_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver fails closed on duplicate daily rollback orphans" \
|
||||
receiver "$DAILY_DUPLICATE_LAYOUT" finalize-test "$SECOND_RUN_ID" "$DAILY_DUPLICATE_SHA"
|
||||
[[ -d "$DAILY_DUPLICATE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.104" \
|
||||
&& -d "$DAILY_DUPLICATE_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.105" ]]
|
||||
pass "receiver preserves duplicate daily rollback evidence without choosing one"
|
||||
|
||||
DAILY_LINK_LAYOUT="$TEST_DIR/receiver-daily-orphan-link"
|
||||
make_layout "$DAILY_LINK_LAYOUT"
|
||||
make_batch "$DAILY_LINK_LAYOUT/archive/$CURRENT_BATCH" "$CURRENT_RUN_ID"
|
||||
ln -s "$DAILY_LINK_LAYOUT/archive/$CURRENT_BATCH" \
|
||||
"$DAILY_LINK_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.106"
|
||||
make_batch "$DAILY_LINK_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID"
|
||||
DAILY_LINK_SHA="$(sha256sum "$DAILY_LINK_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver rejects a symbolic-link daily rollback orphan" \
|
||||
receiver "$DAILY_LINK_LAYOUT" finalize-test "$SECOND_RUN_ID" "$DAILY_LINK_SHA"
|
||||
[[ -L "$DAILY_LINK_LAYOUT/archive/.daily-replacement-${CURRENT_RUN_ID:5:8}.106" \
|
||||
&& -d "$DAILY_LINK_LAYOUT/archive/$CURRENT_BATCH" ]]
|
||||
pass "receiver leaves linked daily rollback evidence untouched"
|
||||
|
||||
WEEKLY_ROLLBACK_LAYOUT="$TEST_DIR/receiver-weekly-orphan-rollback"
|
||||
make_layout "$WEEKLY_ROLLBACK_LAYOUT"
|
||||
make_batch "$WEEKLY_ROLLBACK_LAYOUT/weekly/${CURRENT_WEEK}.rollback.201" "$CURRENT_RUN_ID"
|
||||
WEEKLY_ROLLBACK_SHA="$(make_corrupt_partial "$WEEKLY_ROLLBACK_LAYOUT" "$SECOND_RUN_ID")"
|
||||
expect_failure "receiver restores a weekly rollback when the canonical checkpoint is absent" \
|
||||
receiver "$WEEKLY_ROLLBACK_LAYOUT" finalize-test "$SECOND_RUN_ID" "$WEEKLY_ROLLBACK_SHA"
|
||||
[[ -d "$WEEKLY_ROLLBACK_LAYOUT/weekly/$CURRENT_WEEK" \
|
||||
&& ! -e "$WEEKLY_ROLLBACK_LAYOUT/weekly/${CURRENT_WEEK}.rollback.201" ]]
|
||||
pass "receiver restores the authoritative weekly rollback before the next finalize"
|
||||
|
||||
WEEKLY_COMPLETE_LAYOUT="$TEST_DIR/receiver-weekly-orphan-complete"
|
||||
make_layout "$WEEKLY_COMPLETE_LAYOUT"
|
||||
make_batch "$WEEKLY_COMPLETE_LAYOUT/weekly/$CURRENT_WEEK" "$SECOND_RUN_ID"
|
||||
make_batch "$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.rollback.202" "$CURRENT_RUN_ID"
|
||||
mkdir "$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.new.202"
|
||||
printf 'interrupted staging\n' >"$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.new.202/partial"
|
||||
WEEKLY_COMPLETE_SHA="$(make_corrupt_partial "$WEEKLY_COMPLETE_LAYOUT" "$THIRD_RUN_ID")"
|
||||
expect_failure "receiver cleans weekly rollback and staging after a valid checkpoint was published" \
|
||||
receiver "$WEEKLY_COMPLETE_LAYOUT" finalize-test "$THIRD_RUN_ID" "$WEEKLY_COMPLETE_SHA"
|
||||
[[ -d "$WEEKLY_COMPLETE_LAYOUT/weekly/$CURRENT_WEEK" \
|
||||
&& ! -e "$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.rollback.202" \
|
||||
&& ! -e "$WEEKLY_COMPLETE_LAYOUT/weekly/${CURRENT_WEEK}.new.202" ]]
|
||||
pass "receiver treats weekly staging as non-authoritative after a valid publish"
|
||||
|
||||
WEEKLY_REBUILD_LAYOUT="$TEST_DIR/receiver-weekly-orphan-rebuild"
|
||||
make_layout "$WEEKLY_REBUILD_LAYOUT"
|
||||
make_batch "$WEEKLY_REBUILD_LAYOUT/archive/$CURRENT_BATCH" "$CURRENT_RUN_ID"
|
||||
mkdir "$WEEKLY_REBUILD_LAYOUT/weekly/${CURRENT_WEEK}.new.203"
|
||||
printf 'interrupted staging\n' >"$WEEKLY_REBUILD_LAYOUT/weekly/${CURRENT_WEEK}.new.203/partial"
|
||||
WEEKLY_REBUILD_SHA="$(make_corrupt_partial "$WEEKLY_REBUILD_LAYOUT" "$SECOND_RUN_ID")"
|
||||
expect_failure "receiver rebuilds a missing weekly point from one valid daily archive" \
|
||||
receiver "$WEEKLY_REBUILD_LAYOUT" finalize-test "$SECOND_RUN_ID" "$WEEKLY_REBUILD_SHA"
|
||||
[[ -d "$WEEKLY_REBUILD_LAYOUT/weekly/$CURRENT_WEEK" \
|
||||
&& ! -e "$WEEKLY_REBUILD_LAYOUT/weekly/${CURRENT_WEEK}.new.203" ]]
|
||||
[[ "$(stat -c %i "$WEEKLY_REBUILD_LAYOUT/weekly/$CURRENT_WEEK/database.dump")" \
|
||||
== "$(stat -c %i "$WEEKLY_REBUILD_LAYOUT/archive/$CURRENT_BATCH/database.dump")" ]]
|
||||
pass "receiver rebuilds the weekly checkpoint from the unique valid daily source"
|
||||
|
||||
WEEKLY_AMBIGUOUS_LAYOUT="$TEST_DIR/receiver-weekly-orphan-ambiguous"
|
||||
make_layout "$WEEKLY_AMBIGUOUS_LAYOUT"
|
||||
make_batch "$WEEKLY_AMBIGUOUS_LAYOUT/archive/$CURRENT_BATCH" "$CURRENT_RUN_ID"
|
||||
make_batch "$WEEKLY_AMBIGUOUS_LAYOUT/archive/$SECOND_BATCH" "$SECOND_RUN_ID"
|
||||
mkdir "$WEEKLY_AMBIGUOUS_LAYOUT/weekly/${CURRENT_WEEK}.new.204"
|
||||
make_batch "$WEEKLY_AMBIGUOUS_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID"
|
||||
WEEKLY_AMBIGUOUS_SHA="$(sha256sum "$WEEKLY_AMBIGUOUS_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver fails closed when weekly staging has multiple daily sources" \
|
||||
receiver "$WEEKLY_AMBIGUOUS_LAYOUT" finalize-test "$THIRD_RUN_ID" "$WEEKLY_AMBIGUOUS_SHA"
|
||||
[[ -d "$WEEKLY_AMBIGUOUS_LAYOUT/weekly/${CURRENT_WEEK}.new.204" \
|
||||
&& ! -e "$WEEKLY_AMBIGUOUS_LAYOUT/weekly/$CURRENT_WEEK" ]]
|
||||
pass "receiver preserves ambiguous weekly staging without choosing a daily source"
|
||||
|
||||
WEEKLY_INVALID_LAYOUT="$TEST_DIR/receiver-weekly-orphan-invalid"
|
||||
make_layout "$WEEKLY_INVALID_LAYOUT"
|
||||
make_batch "$WEEKLY_INVALID_LAYOUT/weekly/$CURRENT_WEEK" "$SECOND_RUN_ID"
|
||||
printf 'corruption\n' >>"$WEEKLY_INVALID_LAYOUT/weekly/$CURRENT_WEEK/database.dump"
|
||||
make_batch "$WEEKLY_INVALID_LAYOUT/weekly/${CURRENT_WEEK}.rollback.205" "$CURRENT_RUN_ID"
|
||||
make_batch "$WEEKLY_INVALID_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID"
|
||||
WEEKLY_INVALID_SHA="$(sha256sum "$WEEKLY_INVALID_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
expect_failure "receiver fails closed when a weekly rollback and invalid canonical coexist" \
|
||||
receiver "$WEEKLY_INVALID_LAYOUT" finalize-test "$THIRD_RUN_ID" "$WEEKLY_INVALID_SHA"
|
||||
[[ -d "$WEEKLY_INVALID_LAYOUT/weekly/$CURRENT_WEEK" \
|
||||
&& -d "$WEEKLY_INVALID_LAYOUT/weekly/${CURRENT_WEEK}.rollback.205" ]]
|
||||
pass "receiver preserves both weekly objects when checkpoint state is ambiguous"
|
||||
|
||||
negative_receiver_case() {
|
||||
local label="$1"
|
||||
local mutation="$2"
|
||||
local layout="$TEST_DIR/receiver-negative-${PASS_COUNT}"
|
||||
make_layout "$layout"
|
||||
make_batch "$layout/inbox/${CURRENT_BATCH}.partial" "$CURRENT_RUN_ID"
|
||||
case "$mutation" in
|
||||
corrupt-artifact) printf 'changed\n' >>"$layout/inbox/${CURRENT_BATCH}.partial/database.dump" ;;
|
||||
corrupt-manifest) printf ' ' >>"$layout/inbox/${CURRENT_BATCH}.partial/manifest.json" ;;
|
||||
extra-file) printf 'extra\n' >"$layout/inbox/${CURRENT_BATCH}.partial/extra.txt" ;;
|
||||
symlink) rm "$layout/inbox/${CURRENT_BATCH}.partial/storage.tar.gz"; ln -s database.dump "$layout/inbox/${CURRENT_BATCH}.partial/storage.tar.gz" ;;
|
||||
esac
|
||||
local sha
|
||||
sha="$(sha256sum "$layout/inbox/${CURRENT_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
if [[ "$mutation" == "corrupt-manifest" ]]; then
|
||||
sha="$VALID_MANIFEST_SHA"
|
||||
fi
|
||||
expect_failure "$label" receiver "$layout" finalize-test "$CURRENT_RUN_ID" "$sha"
|
||||
[[ ! -e "$layout/archive/$CURRENT_BATCH" ]]
|
||||
}
|
||||
|
||||
negative_receiver_case "receiver rejects an artifact checksum mismatch" corrupt-artifact
|
||||
negative_receiver_case "receiver rejects a manifest checksum mismatch" corrupt-manifest
|
||||
negative_receiver_case "receiver rejects an extra file" extra-file
|
||||
negative_receiver_case "receiver rejects links in a batch" symlink
|
||||
|
||||
RESUME_LAYOUT="$TEST_DIR/receiver-resume"
|
||||
make_layout "$RESUME_LAYOUT"
|
||||
make_batch "$RESUME_LAYOUT/inbox/${CURRENT_BATCH}.partial" "$CURRENT_RUN_ID"
|
||||
RESUME_MANIFEST_SHA="$(sha256sum "$RESUME_LAYOUT/inbox/${CURRENT_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
mv "$RESUME_LAYOUT/inbox/${CURRENT_BATCH}.partial/storage.tar.gz" "$TEST_DIR/resume-storage.tar.gz"
|
||||
expect_failure "receiver keeps an incomplete partial batch out of archive" \
|
||||
receiver "$RESUME_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$RESUME_MANIFEST_SHA"
|
||||
mv "$TEST_DIR/resume-storage.tar.gz" "$RESUME_LAYOUT/inbox/${CURRENT_BATCH}.partial/storage.tar.gz"
|
||||
receiver "$RESUME_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$RESUME_MANIFEST_SHA" >/dev/null
|
||||
[[ -d "$RESUME_LAYOUT/archive/$CURRENT_BATCH" ]]
|
||||
pass "receiver finalizes the same partial directory after transfer resumes"
|
||||
|
||||
FORCED_LAYOUT="$TEST_DIR/receiver-forced"
|
||||
make_layout "$FORCED_LAYOUT"
|
||||
expect_failure "receiver rejects rsync sender/read mode" env \
|
||||
RECEIVER_CONFIG_FILE="$TEST_DIR/no-receiver-config" \
|
||||
BACKUP_INBOX_DIR="$FORCED_LAYOUT/inbox" BACKUP_ARCHIVE_DIR="$FORCED_LAYOUT/archive" \
|
||||
BACKUP_WEEKLY_DIR="$FORCED_LAYOUT/weekly" \
|
||||
SSH_ORIGINAL_COMMAND="rsync --server --sender -logDtpre.iLsfxCIvu . ." \
|
||||
"$ROOT_DIR/scripts/receive-offhost-backup.sh" forced
|
||||
expect_failure "receiver rejects a sender flag in a later rsync position" env \
|
||||
RECEIVER_CONFIG_FILE="$TEST_DIR/no-receiver-config" \
|
||||
BACKUP_INBOX_DIR="$FORCED_LAYOUT/inbox" BACKUP_ARCHIVE_DIR="$FORCED_LAYOUT/archive" \
|
||||
BACKUP_WEEKLY_DIR="$FORCED_LAYOUT/weekly" \
|
||||
SSH_ORIGINAL_COMMAND="rsync --server -logDtpre.iLsfxCIvu --sender . ." \
|
||||
"$ROOT_DIR/scripts/receive-offhost-backup.sh" forced
|
||||
expect_failure "receiver rejects destructive rsync flags" env \
|
||||
RECEIVER_CONFIG_FILE="$TEST_DIR/no-receiver-config" \
|
||||
BACKUP_INBOX_DIR="$FORCED_LAYOUT/inbox" BACKUP_ARCHIVE_DIR="$FORCED_LAYOUT/archive" \
|
||||
BACKUP_WEEKLY_DIR="$FORCED_LAYOUT/weekly" \
|
||||
SSH_ORIGINAL_COMMAND="rsync --server --delete -logDtpre.iLsfxCIvu . ." \
|
||||
"$ROOT_DIR/scripts/receive-offhost-backup.sh" forced
|
||||
expect_failure "receiver rejects an ordinary shell command" env \
|
||||
RECEIVER_CONFIG_FILE="$TEST_DIR/no-receiver-config" \
|
||||
BACKUP_INBOX_DIR="$FORCED_LAYOUT/inbox" BACKUP_ARCHIVE_DIR="$FORCED_LAYOUT/archive" \
|
||||
BACKUP_WEEKLY_DIR="$FORCED_LAYOUT/weekly" SSH_ORIGINAL_COMMAND="sh -c id" \
|
||||
"$ROOT_DIR/scripts/receive-offhost-backup.sh" forced
|
||||
|
||||
NESTED_LAYOUT="$TEST_DIR/receiver-nested"
|
||||
mkdir -p "$NESTED_LAYOUT/inbox/archive" "$NESTED_LAYOUT/weekly"
|
||||
expect_failure "receiver rejects archive nested inside the rsync inbox" env \
|
||||
RECEIVER_CONFIG_FILE="$TEST_DIR/no-receiver-config" \
|
||||
BACKUP_INBOX_DIR="$NESTED_LAYOUT/inbox" BACKUP_ARCHIVE_DIR="$NESTED_LAYOUT/inbox/archive" \
|
||||
BACKUP_WEEKLY_DIR="$NESTED_LAYOUT/weekly" \
|
||||
"$ROOT_DIR/scripts/receive-offhost-backup.sh" validate-config
|
||||
|
||||
RETENTION_LAYOUT="$TEST_DIR/receiver-retention"
|
||||
make_layout "$RETENTION_LAYOUT"
|
||||
for day in 01 02 03 04 05 06 07; do
|
||||
mkdir "$RETENTION_LAYOUT/archive/hyb-oa-backup-auto-202601${day}T023000Z"
|
||||
done
|
||||
for week in 01 02 03 04; do
|
||||
mkdir "$RETENTION_LAYOUT/weekly/week-2026-W${week}"
|
||||
done
|
||||
make_batch "$RETENTION_LAYOUT/inbox/${CURRENT_BATCH}.partial" "$CURRENT_RUN_ID"
|
||||
RETENTION_SHA="$(sha256sum "$RETENTION_LAYOUT/inbox/${CURRENT_BATCH}.partial/manifest.json" | awk '{print $1}')"
|
||||
receiver "$RETENTION_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$RETENTION_SHA" >/dev/null
|
||||
[[ "$(find "$RETENTION_LAYOUT/archive" -mindepth 1 -maxdepth 1 -type d | wc -l)" == "7" ]]
|
||||
[[ "$(find "$RETENTION_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d | wc -l)" == "4" ]]
|
||||
[[ -d "$RETENTION_LAYOUT/archive/$CURRENT_BATCH" ]]
|
||||
pass "receiver retains seven daily and four weekly recovery points while protecting current success"
|
||||
|
||||
EMPTY_SOURCE="$TEST_DIR/empty-rsync-source"
|
||||
EMPTY_TARGET="$TEST_DIR/empty-rsync-target"
|
||||
mkdir "$EMPTY_SOURCE" "$EMPTY_TARGET"
|
||||
rsync -a --dry-run --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= "$EMPTY_SOURCE/" "$EMPTY_TARGET/"
|
||||
pass "installed rsync accepts the production chmod contract"
|
||||
|
||||
WRAPPER_ROOT="$TEST_DIR/wrapper-root"
|
||||
WRAPPER_PG="$TEST_DIR/wrapper-pg"
|
||||
WRAPPER_STORAGE="$TEST_DIR/wrapper-storage"
|
||||
WRAPPER_BACKUPS="$TEST_DIR/wrapper-backups"
|
||||
WRAPPER_STATE="$TEST_DIR/wrapper-state"
|
||||
MOCK_BIN="$TEST_DIR/mock-bin"
|
||||
mkdir -p "$WRAPPER_ROOT/deploy/trialrun" "$WRAPPER_PG" "$WRAPPER_STORAGE" \
|
||||
"$WRAPPER_BACKUPS" "$WRAPPER_STATE" "$MOCK_BIN"
|
||||
printf '16\n' >"$WRAPPER_PG/PG_VERSION"
|
||||
printf 'synthetic attachment\n' >"$WRAPPER_STORAGE/marker.txt"
|
||||
printf 'synthetic env\n' >"$WRAPPER_ROOT/deploy/trialrun/.env"
|
||||
ln -s "$ROOT_DIR/scripts" "$WRAPPER_ROOT/scripts"
|
||||
printf 'private-key-fixture\n' >"$TEST_DIR/offhost-key"
|
||||
chmod 0600 "$TEST_DIR/offhost-key"
|
||||
printf 'backup-alias ssh-ed25519 fixture\n' >"$TEST_DIR/known-hosts"
|
||||
|
||||
BUILD_SHA="abcdef1234567890abcdef1234567890abcdef12"
|
||||
VITE_BUILD_SHA="${BUILD_SHA:0:12}"
|
||||
MOCK_ALERT_LOG="$TEST_DIR/mock-alert.log"
|
||||
MOCK_DOCKER_ARGV_LOG="$TEST_DIR/mock-docker-argv.log"
|
||||
MOCK_DOCKER_ENV_LOG="$TEST_DIR/mock-docker-env.log"
|
||||
export MOCK_PG="$WRAPPER_PG" MOCK_STORAGE="$WRAPPER_STORAGE" MOCK_BUILD_SHA="$BUILD_SHA" MOCK_ALERT_LOG
|
||||
export MOCK_DOCKER_ARGV_LOG MOCK_DOCKER_ENV_LOG
|
||||
|
||||
cat >"$MOCK_BIN/docker" <<'MOCK'
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
if [[ "${1:-}" == "compose" && "${2:-}" == "version" ]]; then exit 0; fi
|
||||
if [[ "${1:-}" == "compose" ]]; then
|
||||
args=" $* "
|
||||
if [[ "$args" == *" ps -q server "* ]]; then echo server-id; exit 0; fi
|
||||
if [[ "$args" == *" ps -q db "* ]]; then echo db-id; exit 0; fi
|
||||
if [[ "$args" == *" exec -T server "* ]]; then exit 99; fi
|
||||
fi
|
||||
if [[ "${1:-}" == "inspect" ]]; then
|
||||
template="${3:-}"
|
||||
container="${4:-}"
|
||||
case "$template" in
|
||||
*State.Status*) echo running ;;
|
||||
*Config.Env*)
|
||||
if [[ "$container" == "server-id" ]]; then
|
||||
printf 'BUILD_SHA=%s\nSTORAGE_DIR=/app/storage\nDATABASE_URL=postgresql://backup_user:fixture@db:5432/hyb_oa\n' "$MOCK_BUILD_SHA"
|
||||
else
|
||||
printf 'POSTGRES_DB=hyb_oa\nPOSTGRES_USER=backup_user\nPOSTGRES_PASSWORD=fixture\n'
|
||||
fi
|
||||
;;
|
||||
*json*.Mounts*)
|
||||
if [[ "$container" == "server-id" ]]; then
|
||||
printf '[{"Destination":"/app/storage","Source":"%s"}]\n' "$MOCK_STORAGE"
|
||||
else
|
||||
printf '[{"Destination":"/var/lib/postgresql/data","Source":"%s"}]\n' "$MOCK_PG"
|
||||
fi
|
||||
;;
|
||||
*Image*) printf 'sha256:%064d\n' 0 ;;
|
||||
*) exit 1 ;;
|
||||
esac
|
||||
exit 0
|
||||
fi
|
||||
if [[ "${1:-}" == "exec" ]]; then
|
||||
printf '%q ' "$@" >>"$MOCK_DOCKER_ARGV_LOG"
|
||||
printf '\n' >>"$MOCK_DOCKER_ARGV_LOG"
|
||||
printf 'PGPASSWORD=%s\n' "${PGPASSWORD:-}" >>"$MOCK_DOCKER_ENV_LOG"
|
||||
if [[ " $* " == *" pg_dump "* ]]; then printf 'synthetic-dump\n'; exit 0; fi
|
||||
if [[ " $* " == *" psql "* ]]; then printf '%s\n' "${POSTGRES_DB:-hyb_oa_restore_shadow}"; exit 0; fi
|
||||
if [[ " $* " == *" pg_restore --list "* ]]; then cat >/dev/null; exit 0; fi
|
||||
if [[ " $* " == *" pg_restore "* ]]; then cat >/dev/null; exit 0; fi
|
||||
fi
|
||||
exit 1
|
||||
MOCK
|
||||
|
||||
cat >"$MOCK_BIN/rsync" <<'MOCK'
|
||||
#!/usr/bin/env bash
|
||||
echo 'synthetic transfer diagnostic' >&2
|
||||
exit "${MOCK_RSYNC_EXIT:-1}"
|
||||
MOCK
|
||||
cat >"$MOCK_BIN/ssh" <<'MOCK'
|
||||
#!/usr/bin/env bash
|
||||
exit "${MOCK_SSH_EXIT:-0}"
|
||||
MOCK
|
||||
chmod +x "$MOCK_BIN/docker" "$MOCK_BIN/rsync" "$MOCK_BIN/ssh"
|
||||
WECOM_TEST_HOST='qyapi.weixin.qq.com'
|
||||
WECOM_TEST_PATH='/cgi-bin/webhook/send'
|
||||
printf 'https://%s%s?key=%s\n' "$WECOM_TEST_HOST" "$WECOM_TEST_PATH" \
|
||||
'00000000-0000-0000-0000-000000000000' >"$TEST_DIR/wecom-webhook"
|
||||
chmod 0600 "$TEST_DIR/wecom-webhook"
|
||||
ALERT_CAPTURE="$TEST_DIR/alert-capture.jsonl"
|
||||
cat >"$TEST_DIR/failure-summary.json" <<'JSON'
|
||||
{"issueId":"AUTO_BACKUP","status":"FAIL_AUTOMATIC_BACKUP","conclusion":"FAIL_AUTOMATIC_BACKUP","findingCount":1,"findings":[{"source":"automatic_backup","status":"FAIL_AUTOMATIC_BACKUP","severity":"P0","detail":"synthetic_failure"}]}
|
||||
JSON
|
||||
chmod 0600 "$TEST_DIR/failure-summary.json"
|
||||
chmod 0644 "$TEST_DIR/wecom-webhook"
|
||||
expect_failure "group alert helper rejects a webhook file that is not mode 0600" env \
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \
|
||||
python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook" \
|
||||
--summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json"
|
||||
chmod 0600 "$TEST_DIR/wecom-webhook"
|
||||
ln -s "$TEST_DIR/wecom-webhook" "$TEST_DIR/wecom-webhook-link"
|
||||
expect_failure "group alert helper rejects a symbolic-link webhook file" env \
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \
|
||||
python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook-link" \
|
||||
--summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json"
|
||||
printf 'https://%s%s?key=%s&extra=forbidden\n' "$WECOM_TEST_HOST" "$WECOM_TEST_PATH" \
|
||||
'00000000-0000-0000-0000-000000000000' >"$TEST_DIR/wecom-webhook-invalid"
|
||||
chmod 0600 "$TEST_DIR/wecom-webhook-invalid"
|
||||
expect_failure "group alert helper rejects a webhook URL with an unexpected query field" env \
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \
|
||||
python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook-invalid" \
|
||||
--summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json"
|
||||
|
||||
wrapper_env=(
|
||||
env PATH="$MOCK_BIN:$PATH" REMOTE_DIR="$WRAPPER_ROOT"
|
||||
REMOTE_ENV_FILE="$WRAPPER_ROOT/deploy/trialrun/.env"
|
||||
COMPOSE_PROJECT_NAME="hyb-oa-current" POSTGRES_DATA_DIR="$WRAPPER_PG"
|
||||
STORAGE_HOST_DIR="$WRAPPER_STORAGE" BUILD_SHA="$BUILD_SHA" VITE_BUILD_SHA="$VITE_BUILD_SHA"
|
||||
AUTOMATIC_BACKUP_DIR="$WRAPPER_BACKUPS" AUTOMATIC_BACKUP_STATE_DIR="$WRAPPER_STATE"
|
||||
OFFHOST_TARGET="hybbackup@backup-alias" OFFHOST_SSH_KEY="$TEST_DIR/offhost-key"
|
||||
OFFHOST_KNOWN_HOSTS="$TEST_DIR/known-hosts" ALERT_WEBHOOK_FILE="$TEST_DIR/wecom-webhook"
|
||||
ALERT_HELPER="$ROOT_DIR/scripts/send-wecom-group-alert.py"
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE"
|
||||
)
|
||||
|
||||
exec 8>"$WRAPPER_STATE/automatic-backup.lock"
|
||||
flock -n 8
|
||||
expect_failure "wrapper rejects a concurrent automatic-backup run" \
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" run
|
||||
flock -u 8
|
||||
|
||||
rm -f "$WRAPPER_STATE/last-success.json"
|
||||
for day in 01 02 03 04 05 06 07; do
|
||||
mkdir "$WRAPPER_BACKUPS/hyb-oa-backup-auto-202601${day}T023000Z"
|
||||
done
|
||||
expect_failure "unreachable rsync fails without writing off-host success" \
|
||||
"${wrapper_env[@]}" MOCK_RSYNC_EXIT=1 "$ROOT_DIR/scripts/run-automatic-backup.sh" run
|
||||
[[ ! -e "$WRAPPER_STATE/last-success.json" ]]
|
||||
[[ "$(find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' | wc -l)" == "8" ]]
|
||||
pass "source retention does not run before off-host finalize succeeds"
|
||||
python3 - "$WRAPPER_STATE/last-run.json" <<'PYTHON'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
serialized = json.dumps(payload)
|
||||
assert payload["issueId"] == "AUTO_BACKUP"
|
||||
assert payload["status"] == "FAIL_AUTOMATIC_BACKUP"
|
||||
assert not any(value in serialized for value in ("/tmp/", "postgresql://", "backup-alias", "fixture"))
|
||||
PYTHON
|
||||
pass "wrapper failure summary is category-only and redacted"
|
||||
[[ -f "$WRAPPER_STATE/last-error.log" && "$(stat -c %a "$WRAPPER_STATE/last-error.log")" == "600" ]]
|
||||
grep -Fq 'synthetic transfer diagnostic' "$WRAPPER_STATE/last-error.log"
|
||||
pass "wrapper retains the latest command diagnostic locally with mode 0600"
|
||||
[[ -s "$MOCK_DOCKER_ARGV_LOG" && -s "$MOCK_DOCKER_ENV_LOG" ]]
|
||||
! grep -Fq 'fixture' "$MOCK_DOCKER_ARGV_LOG"
|
||||
! grep -Fq 'postgresql://' "$MOCK_DOCKER_ARGV_LOG"
|
||||
grep -Fq 'PGPASSWORD=fixture' "$MOCK_DOCKER_ENV_LOG"
|
||||
pass "wrapper and nightly Docker backup keep connection secrets out of captured argv"
|
||||
|
||||
: >"$MOCK_DOCKER_ARGV_LOG"
|
||||
: >"$MOCK_DOCKER_ENV_LOG"
|
||||
DB_TOOL_MODE=docker DATABASE_URL='postgresql://synthetic:argv-secret@db:5432/hyb_oa_restore_shadow' \
|
||||
DB_CONTAINER=db-id POSTGRES_DB=hyb_oa_restore_shadow POSTGRES_USER=synthetic \
|
||||
POSTGRES_PASSWORD=argv-secret OUT_FILE="$TEST_DIR/argv-backup.dump" PATH="$MOCK_BIN:$PATH" \
|
||||
"$ROOT_DIR/scripts/backup-db.sh" >/dev/null
|
||||
printf 'synthetic restore dump\n' >"$TEST_DIR/argv-restore.dump"
|
||||
DB_TOOL_MODE=docker DATABASE_URL='postgresql://synthetic:argv-secret@db:5432/hyb_oa_restore_shadow' \
|
||||
DB_CONTAINER=db-id POSTGRES_DB=hyb_oa_restore_shadow POSTGRES_USER=synthetic \
|
||||
POSTGRES_PASSWORD=argv-secret RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE=hyb_oa_restore_shadow \
|
||||
PATH="$MOCK_BIN:$PATH" "$ROOT_DIR/scripts/restore-db.sh" "$TEST_DIR/argv-restore.dump" >/dev/null
|
||||
! grep -Fq 'argv-secret' "$MOCK_DOCKER_ARGV_LOG"
|
||||
grep -Fq 'exec -e PGPASSWORD db-id pg_dump' "$MOCK_DOCKER_ARGV_LOG"
|
||||
grep -Fq 'exec -e PGPASSWORD db-id psql' "$MOCK_DOCKER_ARGV_LOG"
|
||||
grep -Fq 'exec -i -e PGPASSWORD db-id pg_restore' "$MOCK_DOCKER_ARGV_LOG"
|
||||
[[ "$(grep -Fc 'PGPASSWORD=argv-secret' "$MOCK_DOCKER_ENV_LOG")" == "3" ]]
|
||||
pass "Docker backup and both restore credential paths inherit PGPASSWORD without argv values"
|
||||
|
||||
! grep -Eq 'capture_command[[:space:]]+paired_backup[[:space:]]+env' "$ROOT_DIR/scripts/run-automatic-backup.sh"
|
||||
! grep -Eq 'docker exec([^\n]|\\\n)*-e[[:space:]]+PGPASSWORD=' \
|
||||
"$ROOT_DIR/scripts/backup-db.sh" "$ROOT_DIR/scripts/restore-db.sh"
|
||||
pass "production backup scripts contain no value-bearing credential argv patterns"
|
||||
|
||||
find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' -exec rm -rf -- {} +
|
||||
for day in 01 02 03 04 05 06 07; do
|
||||
mkdir "$WRAPPER_BACKUPS/hyb-oa-backup-auto-202602${day}T023000Z"
|
||||
done
|
||||
rm -f "$WRAPPER_STATE/last-success.json"
|
||||
expect_failure "off-host finalize rejection fails the source run without publishing success" \
|
||||
"${wrapper_env[@]}" MOCK_RSYNC_EXIT=0 MOCK_SSH_EXIT=1 "$ROOT_DIR/scripts/run-automatic-backup.sh" run
|
||||
[[ ! -e "$WRAPPER_STATE/last-success.json" ]]
|
||||
[[ "$(find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' | wc -l)" == "8" ]]
|
||||
[[ -f "$WRAPPER_STATE/last-error.log" && "$(stat -c %a "$WRAPPER_STATE/last-error.log")" == "600" ]]
|
||||
python3 - "$WRAPPER_STATE/last-run.json" <<'PYTHON'
|
||||
import json
|
||||
import sys
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
assert payload["status"] == "FAIL_AUTOMATIC_BACKUP"
|
||||
assert payload["findings"][0]["detail"] == "phase_offhost_finalize_failed"
|
||||
PYTHON
|
||||
pass "off-host finalize fail-closed preserves the source batch and emits a safe failure category"
|
||||
|
||||
find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' -exec rm -rf -- {} +
|
||||
for day in 01 02 03 04 05 06 07; do
|
||||
mkdir "$WRAPPER_BACKUPS/hyb-oa-backup-auto-202601${day}T023000Z"
|
||||
done
|
||||
"${wrapper_env[@]}" MOCK_RSYNC_EXIT=0 MOCK_SSH_EXIT=0 \
|
||||
"$ROOT_DIR/scripts/run-automatic-backup.sh" run >/dev/null
|
||||
[[ -f "$WRAPPER_STATE/last-success.json" ]]
|
||||
[[ "$(find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' | wc -l)" == "7" ]]
|
||||
pass "off-host finalize success writes state and then prunes source retention to seven"
|
||||
[[ ! -e "$WRAPPER_STATE/last-error.log" ]]
|
||||
pass "wrapper clears the retained diagnostic after a successful off-host run"
|
||||
|
||||
rm -f "$WRAPPER_STATE/last-success.json"
|
||||
expect_failure "stale watchdog rejects a missing success record" \
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
|
||||
|
||||
python3 - "$WRAPPER_STATE/last-success.json" "$BUILD_SHA" <<'PYTHON'
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
with open(sys.argv[1], "w", encoding="utf-8") as handle:
|
||||
json.dump({
|
||||
"schemaVersion": 1,
|
||||
"status": "complete",
|
||||
"completedAt": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
"runId": "auto-20260727T000000Z",
|
||||
"manifestSha256": "0" * 64,
|
||||
"composeProject": "hyb-oa-current",
|
||||
"buildSha": sys.argv[2],
|
||||
}, handle)
|
||||
PYTHON
|
||||
cp "$WRAPPER_STATE/last-success.json" "$WRAPPER_STATE/last-success.valid.json"
|
||||
python3 - "$WRAPPER_STATE/last-success.json" <<'PYTHON'
|
||||
import json
|
||||
import sys
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
payload["buildSha"] = "0" * 40
|
||||
with open(sys.argv[1], "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle)
|
||||
PYTHON
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale >/dev/null
|
||||
python3 - "$WRAPPER_STATE/last-run.json" <<'PYTHON'
|
||||
import json
|
||||
import sys
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
assert payload["status"] == "PASS_WITH_FINDINGS"
|
||||
assert payload["findings"][0]["detail"] == "recovery_point_build_differs"
|
||||
PYTHON
|
||||
pass "stale watchdog downgrades a fresh recovery point from another build to one finding"
|
||||
mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json"
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale >/dev/null
|
||||
pass "stale watchdog accepts a fresh project/build-bound success record"
|
||||
|
||||
cp "$WRAPPER_STATE/last-success.json" "$WRAPPER_STATE/last-success.valid.json"
|
||||
python3 - "$WRAPPER_STATE/last-success.json" <<'PYTHON'
|
||||
import json
|
||||
import sys
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
payload["composeProject"] = "different-project"
|
||||
with open(sys.argv[1], "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle)
|
||||
PYTHON
|
||||
expect_failure "stale watchdog hard-fails a fresh record from another Compose project" \
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
|
||||
mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json"
|
||||
cp "$WRAPPER_STATE/last-success.json" "$WRAPPER_STATE/last-success.valid.json"
|
||||
python3 - "$WRAPPER_STATE/last-success.json" <<'PYTHON'
|
||||
import json
|
||||
import sys
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
payload["manifestSha256"] = "invalid"
|
||||
with open(sys.argv[1], "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle)
|
||||
PYTHON
|
||||
expect_failure "stale watchdog hard-fails a malformed success record" \
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
|
||||
mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json"
|
||||
|
||||
DEADMAN_LAYOUT="$TEST_DIR/deadman"
|
||||
mkdir -p "$DEADMAN_LAYOUT/archive" "$DEADMAN_LAYOUT/state"
|
||||
cp -a "$VALID_LAYOUT/archive/$SECOND_BATCH" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH"
|
||||
DEADMAN_CAPTURE="$TEST_DIR/deadman-alerts.jsonl"
|
||||
deadman_env=(env HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$DEADMAN_CAPTURE")
|
||||
"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null
|
||||
[[ ! -e "$DEADMAN_CAPTURE" ]]
|
||||
pass "ARM dead-man accepts a fresh verified archive without sending a healthy baseline"
|
||||
DEADMAN_CORRUPT_STATE="$TEST_DIR/deadman-corrupt-state"
|
||||
DEADMAN_CORRUPT_CAPTURE="$TEST_DIR/deadman-corrupt-alerts.jsonl"
|
||||
mkdir "$DEADMAN_CORRUPT_STATE"
|
||||
chmod 0600 "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
|
||||
printf 'corruption\n' >>"$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
|
||||
expect_failure "ARM dead-man rejects a corrupt newest finalized archive" env \
|
||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$DEADMAN_CORRUPT_CAPTURE" \
|
||||
python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_CORRUPT_STATE" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py"
|
||||
cp "$VALID_LAYOUT/archive/$SECOND_BATCH/database.dump" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
|
||||
chmod 0400 "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump"
|
||||
pass "ARM dead-man detects archive corruption independently of source state"
|
||||
mv "$DEADMAN_LAYOUT/archive/$SECOND_BATCH" "$TEST_DIR/deadman-withheld-batch"
|
||||
expect_failure "ARM dead-man detects that the source no longer produces a finalized archive" \
|
||||
"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py"
|
||||
[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "1" ]]
|
||||
expect_failure "ARM dead-man suppresses an identical repeated failure alert" \
|
||||
"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py"
|
||||
[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "1" ]]
|
||||
pass "ARM dead-man keeps repeated identical failure alerts deduplicated"
|
||||
mv "$TEST_DIR/deadman-withheld-batch" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH"
|
||||
"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \
|
||||
--archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \
|
||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null
|
||||
[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "2" ]]
|
||||
pass "ARM dead-man sends one recovery notification after finalized archives resume"
|
||||
|
||||
DRIFT_BUILD_SHA="0123456789abcdef0123456789abcdef01234567"
|
||||
expect_failure "wrapper rejects an active runtime build that drifted from its installed contract" \
|
||||
"${wrapper_env[@]}" BUILD_SHA="$DRIFT_BUILD_SHA" VITE_BUILD_SHA="${DRIFT_BUILD_SHA:0:12}" \
|
||||
"$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale
|
||||
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" remind-drill >/dev/null
|
||||
FIRST_REMINDER="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["findings"][0]["detail"])' "$WRAPPER_STATE/last-run.json")"
|
||||
"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" remind-drill >/dev/null
|
||||
SECOND_REMINDER="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["findings"][0]["detail"])' "$WRAPPER_STATE/last-run.json")"
|
||||
[[ "$FIRST_REMINDER" == "$SECOND_REMINDER" && "$FIRST_REMINDER" == quarterly_shadow_restore_due_* ]]
|
||||
pass "quarterly reminder finding key is stable within the same quarter"
|
||||
|
||||
expect_failure "receiver installer uninstall rejects an unsafe target path" env \
|
||||
BACKUP_USER=hybbackup RECEIVER_TARGET=/ CONFIG_FILE=/etc/hyb-oa-test.conf \
|
||||
"$ROOT_DIR/deploy/trialrun/install-offhost-backup-receiver.sh" uninstall
|
||||
|
||||
expect_failure "receiver installer rejects a system-managed backup root before mutation" env \
|
||||
BACKUP_ROOT=/etc/hyb-oa-backups PUBLIC_KEY_FILE="$TEST_DIR/known-hosts" \
|
||||
RECEIVER_SOURCE="$ROOT_DIR/scripts/receive-offhost-backup.sh" \
|
||||
"$ROOT_DIR/deploy/trialrun/install-offhost-backup-receiver.sh" install
|
||||
|
||||
printf '[PASS] automatic off-host backup contract completed (%d checks)\n' "$PASS_COUNT"
|
||||
+558
@@ -0,0 +1,558 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
CONFIG_FILE="${RECEIVER_CONFIG_FILE:-/etc/hyb-oa-offhost-backup.conf}"
|
||||
if [[ -f "$CONFIG_FILE" ]]; then
|
||||
# Root-owned installer output with path-only assignments.
|
||||
# shellcheck disable=SC1090
|
||||
source "$CONFIG_FILE"
|
||||
fi
|
||||
|
||||
BACKUP_INBOX_DIR="${BACKUP_INBOX_DIR:-}"
|
||||
BACKUP_ARCHIVE_DIR="${BACKUP_ARCHIVE_DIR:-}"
|
||||
BACKUP_WEEKLY_DIR="${BACKUP_WEEKLY_DIR:-}"
|
||||
RRSYNC_BIN="${RRSYNC_BIN:-/usr/bin/rrsync}"
|
||||
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||
DAILY_KEEP="${DAILY_KEEP:-7}"
|
||||
WEEKLY_KEEP="${WEEKLY_KEEP:-4}"
|
||||
declare -A RECONCILED_WEEK_SOURCE=()
|
||||
declare -a DAILY_ORPHANS_PENDING_DELETE=()
|
||||
|
||||
fail() {
|
||||
echo "[offhost-receiver][ERROR] $1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
validate_absolute_path() {
|
||||
local name="$1"
|
||||
local value="$2"
|
||||
if [[ -z "$value" || "$value" != /* || ! "$value" =~ ^/[A-Za-z0-9._/-]+$ ]]; then
|
||||
fail "$name must be an explicit safe absolute path"
|
||||
fi
|
||||
if [[ "$value" == "/" || "$value" == */ || "$value" == *"//"* \
|
||||
|| "/$value/" == *"/../"* || "/$value/" == *"/./"* ]]; then
|
||||
fail "$name contains an unsafe path segment"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_layout() {
|
||||
local name value
|
||||
for name in BACKUP_INBOX_DIR BACKUP_ARCHIVE_DIR BACKUP_WEEKLY_DIR; do
|
||||
value="${!name:-}"
|
||||
validate_absolute_path "$name" "$value"
|
||||
[[ -d "$value" && ! -L "$value" ]] || fail "$name is not an existing regular directory"
|
||||
done
|
||||
if [[ "$BACKUP_INBOX_DIR" == "$BACKUP_ARCHIVE_DIR" \
|
||||
|| "$BACKUP_INBOX_DIR" == "$BACKUP_WEEKLY_DIR" \
|
||||
|| "$BACKUP_ARCHIVE_DIR" == "$BACKUP_WEEKLY_DIR" ]]; then
|
||||
fail "backup receiver directories must be distinct"
|
||||
fi
|
||||
local first second
|
||||
for first in "$BACKUP_INBOX_DIR" "$BACKUP_ARCHIVE_DIR" "$BACKUP_WEEKLY_DIR"; do
|
||||
for second in "$BACKUP_INBOX_DIR" "$BACKUP_ARCHIVE_DIR" "$BACKUP_WEEKLY_DIR"; do
|
||||
if [[ "$first" != "$second" && ( "$first" == "$second"/* || "$second" == "$first"/* ) ]]; then
|
||||
fail "backup receiver directories must not be nested"
|
||||
fi
|
||||
done
|
||||
done
|
||||
if [[ "$(stat -c %d "$BACKUP_INBOX_DIR")" != "$(stat -c %d "$BACKUP_ARCHIVE_DIR")" \
|
||||
|| "$(stat -c %d "$BACKUP_ARCHIVE_DIR")" != "$(stat -c %d "$BACKUP_WEEKLY_DIR")" ]]; then
|
||||
fail "receiver directories must share a filesystem for atomic finalize and weekly hardlinks"
|
||||
fi
|
||||
[[ "$DAILY_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "DAILY_KEEP must be positive"
|
||||
[[ "$WEEKLY_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "WEEKLY_KEEP must be positive"
|
||||
command -v "$PYTHON_BIN" >/dev/null 2>&1 || fail "python3 is required"
|
||||
command -v flock >/dev/null 2>&1 || fail "flock is required"
|
||||
}
|
||||
|
||||
validate_run_id() {
|
||||
local run_id="$1"
|
||||
[[ "$run_id" =~ ^auto-[0-9]{8}T[0-9]{6}Z$ ]] \
|
||||
|| fail "run id must use auto-YYYYMMDDTHHMMSSZ"
|
||||
local run_day="${run_id#auto-}"
|
||||
run_day="${run_day%%T*}"
|
||||
[[ "$run_day" == "$(date -u +%Y%m%d)" || "$run_day" == "$(date -u -d yesterday +%Y%m%d)" ]] \
|
||||
|| fail "run id UTC day is outside the accepted receiver window"
|
||||
}
|
||||
|
||||
validate_batch() {
|
||||
local batch_dir="$1"
|
||||
local run_id="$2"
|
||||
local expected_manifest_sha="$3"
|
||||
|
||||
[[ -d "$batch_dir" && ! -L "$batch_dir" ]] || fail "partial batch is missing"
|
||||
[[ "$expected_manifest_sha" =~ ^[0-9a-f]{64}$ ]] || fail "manifest SHA-256 is invalid"
|
||||
|
||||
local entry_count regular_count
|
||||
entry_count="$(find "$batch_dir" -mindepth 1 -maxdepth 1 -printf '.' | wc -c | tr -d ' ')"
|
||||
regular_count="$(find "$batch_dir" -mindepth 1 -maxdepth 1 -type f -printf '.' | wc -c | tr -d ' ')"
|
||||
[[ "$entry_count" == "3" && "$regular_count" == "3" ]] \
|
||||
|| fail "batch must contain exactly three regular files"
|
||||
for file_name in database.dump storage.tar.gz manifest.json; do
|
||||
[[ -f "$batch_dir/$file_name" && ! -L "$batch_dir/$file_name" ]] \
|
||||
|| fail "batch required file is missing"
|
||||
done
|
||||
|
||||
local actual_manifest_sha
|
||||
actual_manifest_sha="$(sha256sum "$batch_dir/manifest.json" | awk '{print $1}')"
|
||||
[[ "$actual_manifest_sha" == "$expected_manifest_sha" ]] \
|
||||
|| fail "manifest checksum mismatch"
|
||||
|
||||
"$PYTHON_BIN" - "$batch_dir" "$run_id" <<'PYTHON' \
|
||||
|| fail "batch manifest or artifact validation failed"
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
batch_dir, expected_run_id = sys.argv[1:]
|
||||
manifest_path = os.path.join(batch_dir, "manifest.json")
|
||||
with open(manifest_path, "r", encoding="utf-8") as handle:
|
||||
manifest = json.load(handle)
|
||||
|
||||
if manifest.get("schemaVersion") != 1:
|
||||
raise SystemExit("unsupported manifest schema")
|
||||
if manifest.get("status") != "complete" or manifest.get("runId") != expected_run_id:
|
||||
raise SystemExit("manifest identity mismatch")
|
||||
if not manifest.get("buildSha") and not manifest.get("sourceImageDigest"):
|
||||
raise SystemExit("manifest source identity missing")
|
||||
|
||||
artifacts = manifest.get("artifacts") or {}
|
||||
expected = {
|
||||
"database": "database.dump",
|
||||
"storage": "storage.tar.gz",
|
||||
}
|
||||
for key, expected_name in expected.items():
|
||||
artifact = artifacts.get(key) or {}
|
||||
if artifact.get("file") != expected_name:
|
||||
raise SystemExit("manifest artifact name mismatch")
|
||||
file_path = os.path.join(batch_dir, expected_name)
|
||||
if os.path.getsize(file_path) != artifact.get("bytes"):
|
||||
raise SystemExit("manifest artifact size mismatch")
|
||||
digest = hashlib.sha256()
|
||||
with open(file_path, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
if digest.hexdigest() != artifact.get("sha256"):
|
||||
raise SystemExit("manifest artifact checksum mismatch")
|
||||
PYTHON
|
||||
}
|
||||
|
||||
validated_batch_identity() {
|
||||
local batch_dir="$1"
|
||||
[[ -d "$batch_dir" && ! -L "$batch_dir" ]] || fail "recovery candidate is not a regular directory"
|
||||
|
||||
local run_id manifest_sha identity
|
||||
run_id="$($PYTHON_BIN - "$batch_dir/manifest.json" <<'PYTHON'
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
|
||||
with open(sys.argv[1], "r", encoding="utf-8") as handle:
|
||||
manifest = json.load(handle)
|
||||
run_id = manifest.get("runId")
|
||||
if not isinstance(run_id, str) or not re.fullmatch(r"auto-[0-9]{8}T[0-9]{6}Z", run_id):
|
||||
raise SystemExit("manifest run id is invalid")
|
||||
print(run_id)
|
||||
PYTHON
|
||||
)" || fail "recovery candidate manifest identity is invalid"
|
||||
manifest_sha="$(sha256sum "$batch_dir/manifest.json" | awk '{print $1}')"
|
||||
validate_batch "$batch_dir" "$run_id" "$manifest_sha"
|
||||
identity="$($PYTHON_BIN - "$run_id" <<'PYTHON'
|
||||
from datetime import datetime
|
||||
import sys
|
||||
|
||||
run_id = sys.argv[1]
|
||||
day = run_id.removeprefix("auto-").split("T", 1)[0]
|
||||
iso = datetime.strptime(day, "%Y%m%d").date().isocalendar()
|
||||
print(f"{run_id}\t{day}\tweek-{iso.year:04d}-W{iso.week:02d}")
|
||||
PYTHON
|
||||
)" || fail "recovery candidate date identity is invalid"
|
||||
printf '%s\t%s\n' "$identity" "$manifest_sha"
|
||||
}
|
||||
|
||||
validate_temp_directory() {
|
||||
local target="$1"
|
||||
local root="$2"
|
||||
local pattern="$3"
|
||||
[[ "$(dirname -- "$target")" == "$root" ]] || fail "orphan target escaped its managed root"
|
||||
[[ "$(basename -- "$target")" =~ $pattern ]] || fail "orphan target name is invalid"
|
||||
[[ -d "$target" && ! -L "$target" ]] || fail "orphan target is not a regular directory"
|
||||
}
|
||||
|
||||
remove_temp_directory() {
|
||||
local target="$1"
|
||||
local root="$2"
|
||||
local pattern="$3"
|
||||
validate_temp_directory "$target" "$root" "$pattern"
|
||||
rm -rf -- "$target"
|
||||
}
|
||||
|
||||
canonical_daily_for_day() {
|
||||
local run_day="$1"
|
||||
local -a entries=()
|
||||
mapfile -t entries < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 \
|
||||
-name "hyb-oa-backup-auto-${run_day}T*Z" -print | LC_ALL=C sort)
|
||||
[[ "${#entries[@]}" -le 1 ]] || fail "daily archive invariant is ambiguous during recovery"
|
||||
if [[ "${#entries[@]}" -eq 1 ]]; then
|
||||
printf '%s\n' "${entries[0]}"
|
||||
fi
|
||||
}
|
||||
|
||||
reconcile_daily_orphans() {
|
||||
local -a orphans=()
|
||||
mapfile -t orphans < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 \
|
||||
-name '.daily-replacement-*' -print | LC_ALL=C sort)
|
||||
local orphan base run_day identity orphan_run_id orphan_day orphan_week orphan_sha
|
||||
local canonical canonical_identity canonical_run_id canonical_day canonical_week canonical_sha
|
||||
local -A seen_days=() identities=() canonicals=()
|
||||
for orphan in "${orphans[@]}"; do
|
||||
base="$(basename -- "$orphan")"
|
||||
[[ "$base" =~ ^\.daily-replacement-([0-9]{8})\.([0-9]+)$ ]] \
|
||||
|| fail "daily replacement orphan name is invalid"
|
||||
run_day="${BASH_REMATCH[1]}"
|
||||
[[ -z "${seen_days[$run_day]:-}" ]] || fail "multiple daily replacement orphans exist for one day"
|
||||
seen_days[$run_day]=1
|
||||
validate_temp_directory "$orphan" "$BACKUP_ARCHIVE_DIR" \
|
||||
'^\.daily-replacement-[0-9]{8}\.[0-9]+$'
|
||||
identity="$(validated_batch_identity "$orphan")" \
|
||||
|| fail "daily replacement orphan validation failed"
|
||||
IFS=$'\t' read -r orphan_run_id orphan_day orphan_week orphan_sha <<<"$identity"
|
||||
[[ "$orphan_day" == "$run_day" ]] || fail "daily replacement orphan day does not match its manifest"
|
||||
identities[$orphan]="$identity"
|
||||
|
||||
canonical="$(canonical_daily_for_day "$run_day")" \
|
||||
|| fail "daily archive lookup failed during recovery"
|
||||
canonicals[$run_day]="$canonical"
|
||||
if [[ -n "$canonical" ]]; then
|
||||
[[ -d "$canonical" && ! -L "$canonical" ]] || fail "daily archive recovery target is invalid"
|
||||
canonical_identity="$(validated_batch_identity "$canonical")" \
|
||||
|| fail "daily archive validation failed during recovery"
|
||||
IFS=$'\t' read -r canonical_run_id canonical_day canonical_week canonical_sha <<<"$canonical_identity"
|
||||
[[ "$canonical_day" == "$run_day" \
|
||||
&& "$(basename -- "$canonical")" == "hyb-oa-backup-$canonical_run_id" ]] \
|
||||
|| fail "daily archive identity is inconsistent during recovery"
|
||||
[[ "$canonical_week" == "$orphan_week" ]] || fail "daily replacement week identity is inconsistent"
|
||||
else
|
||||
canonical="$BACKUP_ARCHIVE_DIR/hyb-oa-backup-$orphan_run_id"
|
||||
[[ ! -e "$canonical" ]] || fail "daily recovery target already exists"
|
||||
fi
|
||||
done
|
||||
|
||||
for orphan in "${orphans[@]}"; do
|
||||
base="$(basename -- "$orphan")"
|
||||
[[ "$base" =~ ^\.daily-replacement-([0-9]{8})\.([0-9]+)$ ]] \
|
||||
|| fail "daily replacement orphan name changed during recovery"
|
||||
run_day="${BASH_REMATCH[1]}"
|
||||
identity="${identities[$orphan]}"
|
||||
IFS=$'\t' read -r orphan_run_id orphan_day orphan_week orphan_sha <<<"$identity"
|
||||
canonical="${canonicals[$run_day]}"
|
||||
if [[ -z "$canonical" ]]; then
|
||||
canonical="$BACKUP_ARCHIVE_DIR/hyb-oa-backup-$orphan_run_id"
|
||||
validate_temp_directory "$orphan" "$BACKUP_ARCHIVE_DIR" \
|
||||
'^\.daily-replacement-[0-9]{8}\.[0-9]+$'
|
||||
[[ ! -e "$canonical" ]] || fail "daily recovery target appeared during recovery"
|
||||
mv -- "$orphan" "$canonical"
|
||||
else
|
||||
DAILY_ORPHANS_PENDING_DELETE+=("$orphan")
|
||||
fi
|
||||
RECONCILED_WEEK_SOURCE[$orphan_week]="$canonical"
|
||||
done
|
||||
}
|
||||
|
||||
prune_named_directories() {
|
||||
local root="$1"
|
||||
local prefix="$2"
|
||||
local keep="$3"
|
||||
local -a entries=()
|
||||
mapfile -t entries < <(find "$root" -mindepth 1 -maxdepth 1 -type d \
|
||||
-name "${prefix}*" -printf '%f\n' | LC_ALL=C sort)
|
||||
local remove_count=$((${#entries[@]} - keep))
|
||||
if (( remove_count <= 0 )); then
|
||||
return 0
|
||||
fi
|
||||
local index entry target
|
||||
for ((index = 0; index < remove_count; index += 1)); do
|
||||
entry="${entries[$index]}"
|
||||
target="$root/$entry"
|
||||
[[ "$entry" == "$prefix"* && -d "$target" && ! -L "$target" ]] \
|
||||
|| fail "retention target failed safety validation"
|
||||
rm -rf -- "$target"
|
||||
done
|
||||
}
|
||||
|
||||
publish_weekly_checkpoint() {
|
||||
local archive_dir="$1"
|
||||
local weekly_dir="$2"
|
||||
local replace_existing="$3"
|
||||
if [[ -e "$weekly_dir" && "$replace_existing" != "1" ]]; then
|
||||
return 0
|
||||
fi
|
||||
local weekly_temp="${weekly_dir}.new.$$"
|
||||
local weekly_rollback="${weekly_dir}.rollback.$$"
|
||||
[[ ! -e "$weekly_temp" && ! -e "$weekly_rollback" ]] || return 1
|
||||
if ! cp -al -- "$archive_dir" "$weekly_temp"; then
|
||||
rm -rf -- "$weekly_temp"
|
||||
return 1
|
||||
fi
|
||||
if [[ -e "$weekly_dir" ]]; then
|
||||
[[ -d "$weekly_dir" && ! -L "$weekly_dir" ]] || return 1
|
||||
mv -- "$weekly_dir" "$weekly_rollback" || return 1
|
||||
fi
|
||||
if ! mv -- "$weekly_temp" "$weekly_dir"; then
|
||||
rm -rf -- "$weekly_temp"
|
||||
if [[ -e "$weekly_rollback" ]]; then
|
||||
mv -- "$weekly_rollback" "$weekly_dir"
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
if [[ -e "$weekly_rollback" ]]; then
|
||||
rm -rf -- "$weekly_rollback"
|
||||
fi
|
||||
}
|
||||
|
||||
unique_daily_source_for_week() {
|
||||
local expected_week="$1"
|
||||
local -a candidates=() entries=()
|
||||
mapfile -t entries < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 \
|
||||
-name 'hyb-oa-backup-auto-*' -print | LC_ALL=C sort)
|
||||
local entry identity run_id run_day iso_week manifest_sha
|
||||
for entry in "${entries[@]}"; do
|
||||
[[ -d "$entry" && ! -L "$entry" ]] || fail "daily archive candidate is invalid during weekly recovery"
|
||||
identity="$(validated_batch_identity "$entry")" \
|
||||
|| fail "daily archive validation failed during weekly recovery"
|
||||
IFS=$'\t' read -r run_id run_day iso_week manifest_sha <<<"$identity"
|
||||
[[ "$(basename -- "$entry")" == "hyb-oa-backup-$run_id" ]] \
|
||||
|| fail "daily archive name does not match its manifest"
|
||||
if [[ "$iso_week" == "$expected_week" ]]; then
|
||||
candidates+=("$entry")
|
||||
fi
|
||||
done
|
||||
[[ "${#candidates[@]}" -eq 1 ]] || fail "weekly recovery source is not unique"
|
||||
printf '%s\n' "${candidates[0]}"
|
||||
}
|
||||
|
||||
reconcile_weekly_orphans() {
|
||||
local -a entries=()
|
||||
mapfile -t entries < <(find "$BACKUP_WEEKLY_DIR" -mindepth 1 -maxdepth 1 \
|
||||
\( -name 'week-*-W*.new.*' -o -name 'week-*-W*.rollback.*' \) -print | LC_ALL=C sort)
|
||||
local -A new_by_week=() rollback_by_week=() weeks=()
|
||||
local entry base week kind
|
||||
for entry in "${entries[@]}"; do
|
||||
base="$(basename -- "$entry")"
|
||||
if [[ "$base" =~ ^(week-[0-9]{4}-W[0-9]{2})\.(new|rollback)\.([0-9]+)$ ]]; then
|
||||
week="${BASH_REMATCH[1]}"
|
||||
kind="${BASH_REMATCH[2]}"
|
||||
else
|
||||
fail "weekly orphan name is invalid"
|
||||
fi
|
||||
validate_temp_directory "$entry" "$BACKUP_WEEKLY_DIR" \
|
||||
'^week-[0-9]{4}-W[0-9]{2}\.(new|rollback)\.[0-9]+$'
|
||||
weeks[$week]=1
|
||||
if [[ "$kind" == "new" ]]; then
|
||||
[[ -z "${new_by_week[$week]:-}" ]] || fail "multiple weekly staging orphans exist"
|
||||
new_by_week[$week]="$entry"
|
||||
else
|
||||
[[ -z "${rollback_by_week[$week]:-}" ]] || fail "multiple weekly rollback orphans exist"
|
||||
rollback_by_week[$week]="$entry"
|
||||
fi
|
||||
done
|
||||
for week in "${!RECONCILED_WEEK_SOURCE[@]}"; do
|
||||
weeks[$week]=1
|
||||
done
|
||||
|
||||
local -A sources=()
|
||||
local canonical new_dir rollback_dir identity run_id run_day iso_week manifest_sha source
|
||||
for week in "${!weeks[@]}"; do
|
||||
canonical="$BACKUP_WEEKLY_DIR/$week"
|
||||
new_dir="${new_by_week[$week]:-}"
|
||||
rollback_dir="${rollback_by_week[$week]:-}"
|
||||
|
||||
if [[ -e "$canonical" ]]; then
|
||||
[[ -d "$canonical" && ! -L "$canonical" ]] || fail "weekly checkpoint is not a regular directory"
|
||||
identity="$(validated_batch_identity "$canonical")" \
|
||||
|| fail "weekly checkpoint validation failed during recovery"
|
||||
IFS=$'\t' read -r run_id run_day iso_week manifest_sha <<<"$identity"
|
||||
[[ "$iso_week" == "$week" ]] || fail "weekly checkpoint manifest belongs to another week"
|
||||
fi
|
||||
if [[ -n "$rollback_dir" ]]; then
|
||||
identity="$(validated_batch_identity "$rollback_dir")" \
|
||||
|| fail "weekly rollback validation failed"
|
||||
IFS=$'\t' read -r run_id run_day iso_week manifest_sha <<<"$identity"
|
||||
[[ "$iso_week" == "$week" ]] || fail "weekly rollback manifest belongs to another week"
|
||||
fi
|
||||
|
||||
source="${RECONCILED_WEEK_SOURCE[$week]:-}"
|
||||
if [[ -z "$source" && ! -e "$canonical" && -z "$rollback_dir" && -n "$new_dir" ]]; then
|
||||
source="$(unique_daily_source_for_week "$week")" \
|
||||
|| fail "weekly staging recovery source could not be resolved"
|
||||
fi
|
||||
if [[ -n "$source" ]]; then
|
||||
[[ -d "$source" && ! -L "$source" ]] || fail "weekly rebuild source is invalid"
|
||||
identity="$(validated_batch_identity "$source")" \
|
||||
|| fail "weekly rebuild source validation failed"
|
||||
IFS=$'\t' read -r run_id run_day iso_week manifest_sha <<<"$identity"
|
||||
[[ "$iso_week" == "$week" ]] || fail "weekly rebuild source belongs to another week"
|
||||
sources[$week]="$source"
|
||||
fi
|
||||
[[ -e "$canonical" || -n "$rollback_dir" || -n "$source" ]] \
|
||||
|| fail "weekly checkpoint has no authoritative recovery source"
|
||||
done
|
||||
|
||||
for week in "${!weeks[@]}"; do
|
||||
canonical="$BACKUP_WEEKLY_DIR/$week"
|
||||
new_dir="${new_by_week[$week]:-}"
|
||||
rollback_dir="${rollback_by_week[$week]:-}"
|
||||
|
||||
if [[ ! -e "$canonical" && -n "$rollback_dir" ]]; then
|
||||
validate_temp_directory "$rollback_dir" "$BACKUP_WEEKLY_DIR" \
|
||||
'^week-[0-9]{4}-W[0-9]{2}\.rollback\.[0-9]+$'
|
||||
mv -- "$rollback_dir" "$canonical"
|
||||
rollback_dir=""
|
||||
fi
|
||||
|
||||
source="${sources[$week]:-}"
|
||||
if [[ -n "$source" ]]; then
|
||||
[[ -d "$source" && ! -L "$source" ]] || fail "weekly rebuild source is invalid"
|
||||
if ! publish_weekly_checkpoint "$source" "$canonical" "$([[ -e "$canonical" ]] && echo 1 || echo 0)"; then
|
||||
fail "weekly checkpoint reconciliation failed"
|
||||
fi
|
||||
fi
|
||||
|
||||
[[ -e "$canonical" ]] || fail "weekly checkpoint is missing after reconciliation"
|
||||
if [[ -n "$rollback_dir" ]]; then
|
||||
remove_temp_directory "$rollback_dir" "$BACKUP_WEEKLY_DIR" \
|
||||
'^week-[0-9]{4}-W[0-9]{2}\.rollback\.[0-9]+$'
|
||||
fi
|
||||
if [[ -n "$new_dir" ]]; then
|
||||
remove_temp_directory "$new_dir" "$BACKUP_WEEKLY_DIR" \
|
||||
'^week-[0-9]{4}-W[0-9]{2}\.new\.[0-9]+$'
|
||||
fi
|
||||
done
|
||||
local pending_daily_orphan
|
||||
for pending_daily_orphan in "${DAILY_ORPHANS_PENDING_DELETE[@]}"; do
|
||||
remove_temp_directory "$pending_daily_orphan" "$BACKUP_ARCHIVE_DIR" \
|
||||
'^\.daily-replacement-[0-9]{8}\.[0-9]+$'
|
||||
done
|
||||
DAILY_ORPHANS_PENDING_DELETE=()
|
||||
RECONCILED_WEEK_SOURCE=()
|
||||
}
|
||||
|
||||
reconcile_interrupted_finalize() {
|
||||
reconcile_daily_orphans
|
||||
reconcile_weekly_orphans
|
||||
}
|
||||
|
||||
finalize_batch() {
|
||||
local run_id="$1"
|
||||
local expected_manifest_sha="$2"
|
||||
validate_run_id "$run_id"
|
||||
validate_layout
|
||||
exec 9>"$BACKUP_ARCHIVE_DIR/.finalize.lock"
|
||||
chmod 0600 "$BACKUP_ARCHIVE_DIR/.finalize.lock"
|
||||
flock -n 9 || fail "another finalize operation is already running"
|
||||
reconcile_interrupted_finalize
|
||||
|
||||
local batch_name="hyb-oa-backup-$run_id"
|
||||
local partial_dir="$BACKUP_INBOX_DIR/${batch_name}.partial"
|
||||
local archive_dir="$BACKUP_ARCHIVE_DIR/$batch_name"
|
||||
local already_archived=0
|
||||
if [[ -e "$archive_dir" ]]; then
|
||||
[[ -d "$archive_dir" && ! -L "$archive_dir" ]] || fail "archive target is invalid"
|
||||
validate_batch "$archive_dir" "$run_id" "$expected_manifest_sha"
|
||||
already_archived=1
|
||||
fi
|
||||
|
||||
local run_day="${run_id#auto-}"
|
||||
run_day="${run_day%%T*}"
|
||||
local existing_for_day="" replacement_rollback=""
|
||||
local -a existing_for_day_entries=()
|
||||
mapfile -t existing_for_day_entries < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 -type d \
|
||||
-name "hyb-oa-backup-auto-${run_day}T*Z" ! -name "$batch_name" -print)
|
||||
[[ "${#existing_for_day_entries[@]}" -le 1 ]] || fail "daily archive invariant is already violated"
|
||||
if [[ "${#existing_for_day_entries[@]}" -eq 1 ]]; then
|
||||
existing_for_day="${existing_for_day_entries[0]}"
|
||||
fi
|
||||
|
||||
if [[ "$already_archived" == "0" ]]; then
|
||||
validate_batch "$partial_dir" "$run_id" "$expected_manifest_sha"
|
||||
chmod 0400 "$partial_dir/database.dump" "$partial_dir/storage.tar.gz" "$partial_dir/manifest.json"
|
||||
if [[ -n "$existing_for_day" ]]; then
|
||||
[[ "$existing_for_day" =~ /hyb-oa-backup-auto-${run_day}T[0-9]{6}Z$ \
|
||||
&& -d "$existing_for_day" && ! -L "$existing_for_day" ]] \
|
||||
|| fail "existing daily archive is invalid"
|
||||
replacement_rollback="$BACKUP_ARCHIVE_DIR/.daily-replacement-${run_day}.$$"
|
||||
[[ ! -e "$replacement_rollback" ]] || fail "daily replacement rollback target already exists"
|
||||
mv -- "$existing_for_day" "$replacement_rollback"
|
||||
fi
|
||||
if ! mv -- "$partial_dir" "$archive_dir"; then
|
||||
if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then
|
||||
mv -- "$replacement_rollback" "$existing_for_day"
|
||||
fi
|
||||
fail "daily archive publication failed"
|
||||
fi
|
||||
fi
|
||||
|
||||
local iso_week
|
||||
iso_week="$($PYTHON_BIN - "$run_day" <<'PYTHON'
|
||||
from datetime import datetime
|
||||
import sys
|
||||
day = datetime.strptime(sys.argv[1], "%Y%m%d").date()
|
||||
iso = day.isocalendar()
|
||||
print(f"week-{iso.year:04d}-W{iso.week:02d}")
|
||||
PYTHON
|
||||
)"
|
||||
local weekly_dir="$BACKUP_WEEKLY_DIR/$iso_week"
|
||||
if ! publish_weekly_checkpoint "$archive_dir" "$weekly_dir" "$([[ -n "$replacement_rollback" ]] && echo 1 || echo 0)"; then
|
||||
if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then
|
||||
rm -rf -- "$archive_dir"
|
||||
mv -- "$replacement_rollback" "$existing_for_day"
|
||||
fi
|
||||
fail "weekly checkpoint update failed"
|
||||
fi
|
||||
if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then
|
||||
rm -rf -- "$replacement_rollback"
|
||||
fi
|
||||
|
||||
prune_named_directories "$BACKUP_ARCHIVE_DIR" "hyb-oa-backup-auto-" "$DAILY_KEEP"
|
||||
prune_named_directories "$BACKUP_WEEKLY_DIR" "week-" "$WEEKLY_KEEP"
|
||||
echo "[offhost-receiver][PASS] finalized batch with verified manifest"
|
||||
}
|
||||
|
||||
run_forced_command() {
|
||||
validate_layout
|
||||
local original_command="${SSH_ORIGINAL_COMMAND:-}"
|
||||
if [[ "$original_command" == rsync\ --server\ * ]]; then
|
||||
if [[ "$original_command" =~ (^|[[:space:]])--sender([[:space:]]|$) ]]; then
|
||||
fail "read access is forbidden"
|
||||
fi
|
||||
if [[ "$original_command" =~ (^|[[:space:]])--(delete|delete-before|delete-after|delete-during|delete-delay|delete-excluded|remove-source-files|remove-sent-files|inplace)(=|[[:space:]]|$) ]]; then
|
||||
fail "destructive rsync option is forbidden"
|
||||
fi
|
||||
[[ -x "$RRSYNC_BIN" ]] || fail "rrsync is required"
|
||||
exec "$RRSYNC_BIN" -wo "$BACKUP_INBOX_DIR"
|
||||
fi
|
||||
|
||||
if [[ "$original_command" =~ ^hyb-backup-finalize[[:space:]](auto-[0-9]{8}T[0-9]{6}Z)[[:space:]]([0-9a-f]{64})$ ]]; then
|
||||
finalize_batch "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}"
|
||||
return 0
|
||||
fi
|
||||
fail "unsupported forced command"
|
||||
}
|
||||
|
||||
case "${1:-forced}" in
|
||||
forced)
|
||||
run_forced_command
|
||||
;;
|
||||
finalize-test)
|
||||
[[ "${RECEIVER_TEST_MODE:-0}" == "1" ]] || fail "test mode is disabled"
|
||||
finalize_batch "${2:-}" "${3:-}"
|
||||
;;
|
||||
validate-config)
|
||||
validate_layout
|
||||
echo "[offhost-receiver][PASS] configuration valid"
|
||||
;;
|
||||
*)
|
||||
fail "unsupported mode"
|
||||
;;
|
||||
esac
|
||||
+2
-2
@@ -98,7 +98,7 @@ if [[ -z "$POSTGRES_DB" || -z "$POSTGRES_USER" || -z "$POSTGRES_PASSWORD" ]]; th
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ACTUAL_DATABASE="$(docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \
|
||||
ACTUAL_DATABASE="$(PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD "$DB_CONTAINER" \
|
||||
psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atqc 'SELECT current_database()')"
|
||||
|
||||
if [[ -z "$ACTUAL_DATABASE" || "$ACTUAL_DATABASE" != "$RESTORE_CONFIRM_DATABASE" ]]; then
|
||||
@@ -110,7 +110,7 @@ validate_shadow_database_name "$ACTUAL_DATABASE"
|
||||
docker exec -i "$DB_CONTAINER" pg_restore --list <"$BACKUP_FILE" >/dev/null
|
||||
|
||||
echo "[INFO] DB restore target confirmed: $ACTUAL_DATABASE (shadow)"
|
||||
docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \
|
||||
PGPASSWORD="$POSTGRES_PASSWORD" docker exec -i -e PGPASSWORD "$DB_CONTAINER" \
|
||||
pg_restore --clean --if-exists --no-owner --no-acl -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
|
||||
<"$BACKUP_FILE"
|
||||
|
||||
|
||||
+477
@@ -0,0 +1,477 @@
|
||||
#!/usr/bin/env bash
|
||||
set -eEuo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
REMOTE_DIR="${REMOTE_DIR:-$ROOT_DIR}"
|
||||
REMOTE_ENV_FILE="${REMOTE_ENV_FILE:-${REMOTE_DIR}/deploy/trialrun/.env}"
|
||||
COMPOSE_FILE="${COMPOSE_FILE:-deploy/trialrun/docker-compose.yml}"
|
||||
COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-}"
|
||||
POSTGRES_DATA_DIR="${POSTGRES_DATA_DIR:-}"
|
||||
STORAGE_HOST_DIR="${STORAGE_HOST_DIR:-}"
|
||||
BUILD_SHA="${BUILD_SHA:-}"
|
||||
VITE_BUILD_SHA="${VITE_BUILD_SHA:-}"
|
||||
AUTOMATIC_BACKUP_DIR="${AUTOMATIC_BACKUP_DIR:-}"
|
||||
AUTOMATIC_BACKUP_STATE_DIR="${AUTOMATIC_BACKUP_STATE_DIR:-}"
|
||||
OFFHOST_TARGET="${OFFHOST_TARGET:-}"
|
||||
OFFHOST_SSH_KEY="${OFFHOST_SSH_KEY:-}"
|
||||
OFFHOST_KNOWN_HOSTS="${OFFHOST_KNOWN_HOSTS:-}"
|
||||
ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
|
||||
ALERT_HELPER="${ALERT_HELPER:-$ROOT_DIR/scripts/send-wecom-group-alert.py}"
|
||||
AUTOMATIC_BACKUP_KEEP="${AUTOMATIC_BACKUP_KEEP:-7}"
|
||||
AUTOMATIC_BACKUP_MAX_AGE_SECONDS="${AUTOMATIC_BACKUP_MAX_AGE_SECONDS:-86400}"
|
||||
MODE="${1:-run}"
|
||||
|
||||
CURRENT_PHASE="preflight"
|
||||
SERVER_CONTAINER=""
|
||||
DB_CONTAINER=""
|
||||
SUMMARY_FILE=""
|
||||
COMPOSE_CMD=()
|
||||
|
||||
fail() {
|
||||
echo "[automatic-backup][ERROR] $1" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
validate_absolute_path() {
|
||||
local name="$1"
|
||||
local value="$2"
|
||||
if [[ -z "$value" || "$value" != /* || ! "$value" =~ ^/[A-Za-z0-9._/-]+$ ]]; then
|
||||
fail "$name must be an explicit safe absolute path"
|
||||
fi
|
||||
if [[ "$value" == "/" || "$value" == */ || "$value" == *"//"* \
|
||||
|| "/$value/" == *"/../"* || "/$value/" == *"/./"* ]]; then
|
||||
fail "$name contains an unsafe path segment"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_managed_directory_path() {
|
||||
local name="$1"
|
||||
local value="$2"
|
||||
validate_absolute_path "$name" "$value"
|
||||
[[ "$value" =~ ^/[^/]+/[^/]+ ]] \
|
||||
|| fail "$name must be a dedicated path below a top-level directory"
|
||||
case "$value" in
|
||||
/bin/*|/boot/*|/dev/*|/etc/*|/lib/*|/lib64/*|/proc/*|/run/*|/sbin/*|/sys/*|/usr/*)
|
||||
fail "$name must not be inside a system-managed directory"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
validate_common() {
|
||||
local name value
|
||||
for name in REMOTE_DIR REMOTE_ENV_FILE POSTGRES_DATA_DIR STORAGE_HOST_DIR \
|
||||
AUTOMATIC_BACKUP_DIR AUTOMATIC_BACKUP_STATE_DIR; do
|
||||
value="${!name:-}"
|
||||
validate_absolute_path "$name" "$value"
|
||||
done
|
||||
validate_managed_directory_path AUTOMATIC_BACKUP_DIR "$AUTOMATIC_BACKUP_DIR"
|
||||
validate_managed_directory_path AUTOMATIC_BACKUP_STATE_DIR "$AUTOMATIC_BACKUP_STATE_DIR"
|
||||
[[ -d "$REMOTE_DIR" && -f "$REMOTE_ENV_FILE" ]] || fail "deployment root or env file is missing"
|
||||
[[ -f "$POSTGRES_DATA_DIR/PG_VERSION" ]] || fail "active PGDATA marker is missing"
|
||||
[[ -d "$STORAGE_HOST_DIR" ]] || fail "active storage directory is missing"
|
||||
[[ "$COMPOSE_PROJECT_NAME" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "Compose project name is invalid"
|
||||
[[ "$COMPOSE_FILE" =~ ^[A-Za-z0-9._/-]+$ && "$COMPOSE_FILE" != /* \
|
||||
&& "/$COMPOSE_FILE/" != *"/../"* && "/$COMPOSE_FILE/" != *"/./"* ]] \
|
||||
|| fail "Compose file must be a safe relative path"
|
||||
[[ "$BUILD_SHA" =~ ^[0-9a-fA-F]{40,64}$ ]] || fail "full build SHA is invalid"
|
||||
BUILD_SHA="${BUILD_SHA,,}"
|
||||
VITE_BUILD_SHA="${VITE_BUILD_SHA,,}"
|
||||
[[ "$VITE_BUILD_SHA" =~ ^[0-9a-f]{12}$ && "${BUILD_SHA:0:12}" == "$VITE_BUILD_SHA" ]] \
|
||||
|| fail "frontend build SHA must match the full build prefix"
|
||||
[[ "$AUTOMATIC_BACKUP_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "backup retention must be positive"
|
||||
[[ "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" =~ ^[1-9][0-9]*$ ]] || fail "stale threshold must be positive"
|
||||
validate_absolute_path ALERT_WEBHOOK_FILE "$ALERT_WEBHOOK_FILE"
|
||||
validate_absolute_path ALERT_HELPER "$ALERT_HELPER"
|
||||
[[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
|
||||
&& "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" ]] \
|
||||
|| fail "alert webhook file must exist with mode 0600"
|
||||
[[ -f "$ALERT_HELPER" && ! -L "$ALERT_HELPER" ]] || fail "alert helper is missing"
|
||||
if [[ "$AUTOMATIC_BACKUP_DIR" == "$AUTOMATIC_BACKUP_STATE_DIR" \
|
||||
|| "$AUTOMATIC_BACKUP_DIR" == "$AUTOMATIC_BACKUP_STATE_DIR"/* \
|
||||
|| "$AUTOMATIC_BACKUP_STATE_DIR" == "$AUTOMATIC_BACKUP_DIR"/* ]]; then
|
||||
fail "backup and state directories must be independent"
|
||||
fi
|
||||
mkdir -p "$AUTOMATIC_BACKUP_DIR" "$AUTOMATIC_BACKUP_STATE_DIR"
|
||||
[[ -d "$AUTOMATIC_BACKUP_DIR" && ! -L "$AUTOMATIC_BACKUP_DIR" \
|
||||
&& -d "$AUTOMATIC_BACKUP_STATE_DIR" && ! -L "$AUTOMATIC_BACKUP_STATE_DIR" ]] \
|
||||
|| fail "backup and state directories must not be symbolic links"
|
||||
chmod 0700 "$AUTOMATIC_BACKUP_DIR" "$AUTOMATIC_BACKUP_STATE_DIR"
|
||||
command -v python3 >/dev/null 2>&1 || fail "python3 is required"
|
||||
}
|
||||
|
||||
validate_transfer_config() {
|
||||
validate_absolute_path OFFHOST_SSH_KEY "$OFFHOST_SSH_KEY"
|
||||
validate_absolute_path OFFHOST_KNOWN_HOSTS "$OFFHOST_KNOWN_HOSTS"
|
||||
[[ -f "$OFFHOST_SSH_KEY" && ! -L "$OFFHOST_SSH_KEY" ]] || fail "off-host private key is missing"
|
||||
[[ "$(stat -c %a "$OFFHOST_SSH_KEY")" == "600" ]] || fail "off-host private key mode must be 0600"
|
||||
[[ -f "$OFFHOST_KNOWN_HOSTS" && -s "$OFFHOST_KNOWN_HOSTS" ]] || fail "pinned known-hosts file is missing"
|
||||
[[ "$OFFHOST_TARGET" =~ ^[a-z_][a-z0-9_-]*@[A-Za-z0-9._-]+$ ]] \
|
||||
|| fail "off-host target must use a safe user@alias value"
|
||||
command -v ssh >/dev/null 2>&1 || fail "ssh is required"
|
||||
command -v rsync >/dev/null 2>&1 || fail "rsync is required"
|
||||
}
|
||||
|
||||
select_compose() {
|
||||
if docker compose version >/dev/null 2>&1; then
|
||||
COMPOSE_CMD=(docker compose)
|
||||
elif command -v docker-compose >/dev/null 2>&1; then
|
||||
COMPOSE_CMD=(docker-compose)
|
||||
else
|
||||
fail "Compose runtime is unavailable"
|
||||
fi
|
||||
}
|
||||
|
||||
compose() {
|
||||
POSTGRES_DATA_DIR="$POSTGRES_DATA_DIR" \
|
||||
STORAGE_HOST_DIR="$STORAGE_HOST_DIR" \
|
||||
BUILD_SHA="$BUILD_SHA" \
|
||||
VITE_BUILD_SHA="$VITE_BUILD_SHA" \
|
||||
"${COMPOSE_CMD[@]}" -p "$COMPOSE_PROJECT_NAME" \
|
||||
-f "$REMOTE_DIR/$COMPOSE_FILE" --env-file "$REMOTE_ENV_FILE" "$@"
|
||||
}
|
||||
|
||||
container_env() {
|
||||
local container_id="$1"
|
||||
local key="$2"
|
||||
docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' "$container_id" \
|
||||
| sed -n "s/^${key}=//p" | head -n 1
|
||||
}
|
||||
|
||||
mount_source() {
|
||||
local container_id="$1"
|
||||
local destination="$2"
|
||||
docker inspect -f '{{json .Mounts}}' "$container_id" \
|
||||
| python3 -c '
|
||||
import json
|
||||
import sys
|
||||
|
||||
destination = sys.argv[1]
|
||||
mounts = json.load(sys.stdin)
|
||||
matches = [mount.get("Source", "") for mount in mounts if mount.get("Destination") == destination]
|
||||
if len(matches) != 1 or not matches[0]:
|
||||
raise SystemExit(1)
|
||||
print(matches[0])
|
||||
' "$destination"
|
||||
}
|
||||
|
||||
resolve_active_runtime() {
|
||||
select_compose
|
||||
local -a servers=() dbs=()
|
||||
mapfile -t servers < <(compose ps -q server)
|
||||
mapfile -t dbs < <(compose ps -q db)
|
||||
[[ "${#servers[@]}" -eq 1 && -n "${servers[0]}" ]] || fail "expected exactly one active server container"
|
||||
[[ "${#dbs[@]}" -eq 1 && -n "${dbs[0]}" ]] || fail "expected exactly one active database container"
|
||||
SERVER_CONTAINER="${servers[0]}"
|
||||
DB_CONTAINER="${dbs[0]}"
|
||||
[[ "$(docker inspect -f '{{.State.Status}}' "$SERVER_CONTAINER")" == "running" ]] \
|
||||
|| fail "server container is not running"
|
||||
[[ "$(docker inspect -f '{{.State.Status}}' "$DB_CONTAINER")" == "running" ]] \
|
||||
|| fail "database container is not running"
|
||||
|
||||
local actual_build storage_destination actual_storage actual_pgdata
|
||||
actual_build="$(container_env "$SERVER_CONTAINER" BUILD_SHA)"
|
||||
storage_destination="$(container_env "$SERVER_CONTAINER" STORAGE_DIR)"
|
||||
[[ -n "$storage_destination" && "$storage_destination" == /* ]] || fail "server storage destination is invalid"
|
||||
actual_storage="$(mount_source "$SERVER_CONTAINER" "$storage_destination")"
|
||||
actual_pgdata="$(mount_source "$DB_CONTAINER" /var/lib/postgresql/data)"
|
||||
[[ "$actual_build" == "$BUILD_SHA" ]] || fail "active server build does not match the installed backup contract"
|
||||
[[ "$actual_storage" == "$STORAGE_HOST_DIR" ]] || fail "active storage mount does not match the installed backup contract"
|
||||
[[ "$actual_pgdata" == "$POSTGRES_DATA_DIR" ]] || fail "active PGDATA mount does not match the installed backup contract"
|
||||
}
|
||||
|
||||
write_summary() {
|
||||
local status="$1"
|
||||
local finding_count="$2"
|
||||
local detail="$3"
|
||||
local output_file="$AUTOMATIC_BACKUP_STATE_DIR/last-run.json"
|
||||
local temp_file="$AUTOMATIC_BACKUP_STATE_DIR/.last-run.json.tmp.$$"
|
||||
python3 - "$temp_file" "$status" "$finding_count" "$detail" "$VITE_BUILD_SHA" <<'PYTHON'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
file_path, status, finding_count, detail, build_sha = sys.argv[1:]
|
||||
finding_count = int(finding_count)
|
||||
findings = []
|
||||
if finding_count:
|
||||
findings.append({
|
||||
"source": "automatic_backup",
|
||||
"status": status,
|
||||
"severity": "P0",
|
||||
"detail": detail,
|
||||
})
|
||||
payload = {
|
||||
"issueId": "AUTO_BACKUP",
|
||||
"status": status,
|
||||
"conclusion": status,
|
||||
"findingCount": finding_count,
|
||||
"findings": findings,
|
||||
"generatedAt": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
"reportDir": "host-backup-state",
|
||||
"buildSha": build_sha,
|
||||
"privacy": "status/count/category only; no credentials, host addresses, absolute paths, file names, or payloads",
|
||||
}
|
||||
with open(file_path, "x", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, ensure_ascii=True, indent=2)
|
||||
handle.write("\n")
|
||||
os.chmod(file_path, 0o600)
|
||||
PYTHON
|
||||
mv -f -- "$temp_file" "$output_file"
|
||||
SUMMARY_FILE="$output_file"
|
||||
}
|
||||
|
||||
send_summary_alert() {
|
||||
local state_name="$1"
|
||||
[[ -n "$SUMMARY_FILE" && -f "$SUMMARY_FILE" ]] || return 1
|
||||
python3 "$ALERT_HELPER" \
|
||||
--webhook-file "$ALERT_WEBHOOK_FILE" \
|
||||
--summary-file "$SUMMARY_FILE" \
|
||||
--state-file "$AUTOMATIC_BACKUP_STATE_DIR/$state_name" >/dev/null
|
||||
}
|
||||
|
||||
capture_command() {
|
||||
local phase="$1"
|
||||
shift
|
||||
local temporary="$AUTOMATIC_BACKUP_STATE_DIR/.${phase}.diagnostic.$$"
|
||||
: >"$temporary"
|
||||
chmod 0600 "$temporary"
|
||||
if "$@" >"$temporary" 2>&1; then
|
||||
rm -f -- "$temporary"
|
||||
return 0
|
||||
fi
|
||||
mv -f -- "$temporary" "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log"
|
||||
chmod 0600 "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log"
|
||||
return 1
|
||||
}
|
||||
|
||||
on_error() {
|
||||
local exit_code="$?"
|
||||
trap - ERR
|
||||
set +e
|
||||
write_summary "FAIL_AUTOMATIC_BACKUP" 1 "phase_${CURRENT_PHASE}_failed"
|
||||
send_summary_alert "automatic-backup-alert-state.json" || true
|
||||
echo "[automatic-backup][FAIL] phase=${CURRENT_PHASE}" >&2
|
||||
exit "$exit_code"
|
||||
}
|
||||
|
||||
prune_local_backups() {
|
||||
local current_name="$1"
|
||||
local -a entries=()
|
||||
mapfile -t entries < <(find "$AUTOMATIC_BACKUP_DIR" -mindepth 1 -maxdepth 1 -type d \
|
||||
-name 'hyb-oa-backup-auto-*' -printf '%f\n' | LC_ALL=C sort)
|
||||
local remove_count=$((${#entries[@]} - AUTOMATIC_BACKUP_KEEP))
|
||||
if (( remove_count <= 0 )); then
|
||||
return 0
|
||||
fi
|
||||
local index entry target
|
||||
for ((index = 0; index < remove_count; index += 1)); do
|
||||
entry="${entries[$index]}"
|
||||
[[ "$entry" != "$current_name" && "$entry" =~ ^hyb-oa-backup-auto-[0-9]{8}T[0-9]{6}Z$ ]] \
|
||||
|| fail "local retention target failed safety validation"
|
||||
target="$AUTOMATIC_BACKUP_DIR/$entry"
|
||||
[[ -d "$target" && ! -L "$target" ]] || fail "local retention target is invalid"
|
||||
rm -rf -- "$target"
|
||||
done
|
||||
}
|
||||
|
||||
write_success_state() {
|
||||
local run_id="$1"
|
||||
local manifest_sha="$2"
|
||||
local temp_file="$AUTOMATIC_BACKUP_STATE_DIR/.last-success.json.tmp.$$"
|
||||
python3 - "$temp_file" "$run_id" "$manifest_sha" "$COMPOSE_PROJECT_NAME" "$BUILD_SHA" <<'PYTHON'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
file_path, run_id, manifest_sha, project, build_sha = sys.argv[1:]
|
||||
payload = {
|
||||
"schemaVersion": 1,
|
||||
"status": "complete",
|
||||
"completedAt": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
"runId": run_id,
|
||||
"manifestSha256": manifest_sha,
|
||||
"composeProject": project,
|
||||
"buildSha": build_sha,
|
||||
}
|
||||
with open(file_path, "x", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, ensure_ascii=True, indent=2)
|
||||
handle.write("\n")
|
||||
os.chmod(file_path, 0o600)
|
||||
PYTHON
|
||||
mv -f -- "$temp_file" "$AUTOMATIC_BACKUP_STATE_DIR/last-success.json"
|
||||
}
|
||||
|
||||
run_backup() {
|
||||
validate_common
|
||||
validate_transfer_config
|
||||
exec 9>"$AUTOMATIC_BACKUP_STATE_DIR/automatic-backup.lock"
|
||||
flock -n 9 || fail "another automatic backup is already running"
|
||||
trap on_error ERR
|
||||
|
||||
CURRENT_PHASE="runtime_identity"
|
||||
resolve_active_runtime
|
||||
|
||||
local database_url postgres_db postgres_user postgres_password parsed_identity url_user url_db
|
||||
database_url="$(container_env "$SERVER_CONTAINER" DATABASE_URL)"
|
||||
postgres_db="$(container_env "$DB_CONTAINER" POSTGRES_DB)"
|
||||
postgres_user="$(container_env "$DB_CONTAINER" POSTGRES_USER)"
|
||||
postgres_password="$(container_env "$DB_CONTAINER" POSTGRES_PASSWORD)"
|
||||
[[ -n "$database_url" && -n "$postgres_db" && -n "$postgres_user" && -n "$postgres_password" ]] \
|
||||
|| fail "active database identity is incomplete"
|
||||
parsed_identity="$(printf '%s' "$database_url" | python3 -c '
|
||||
import sys
|
||||
from urllib.parse import unquote, urlparse
|
||||
value = urlparse(sys.stdin.read())
|
||||
if value.scheme not in {"postgres", "postgresql"} or not value.username or not value.path.strip("/"):
|
||||
raise SystemExit(1)
|
||||
database_name = unquote(value.path.strip("/"))
|
||||
print(f"{unquote(value.username)}\t{database_name}")
|
||||
')"
|
||||
IFS=$'\t' read -r url_user url_db <<<"$parsed_identity"
|
||||
[[ "$url_user" == "$postgres_user" && "$url_db" == "$postgres_db" ]] \
|
||||
|| fail "active server and database container identities do not match"
|
||||
|
||||
local run_id batch_name batch_dir source_image_digest manifest_sha
|
||||
run_id="auto-$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
batch_name="hyb-oa-backup-$run_id"
|
||||
batch_dir="$AUTOMATIC_BACKUP_DIR/$batch_name"
|
||||
source_image_digest="$(docker inspect -f '{{.Image}}' "$SERVER_CONTAINER")"
|
||||
[[ "$source_image_digest" =~ ^sha256:[0-9a-f]{64}$ ]] || fail "active server image digest is invalid"
|
||||
|
||||
CURRENT_PHASE="paired_backup"
|
||||
if ! (
|
||||
export DATABASE_URL="$database_url"
|
||||
export DB_TOOL_MODE=docker DB_CONTAINER
|
||||
export POSTGRES_DB="$postgres_db" POSTGRES_USER="$postgres_user"
|
||||
export POSTGRES_PASSWORD="$postgres_password"
|
||||
export STORAGE_DIR_HOST="$STORAGE_HOST_DIR" BACKUP_DIR="$AUTOMATIC_BACKUP_DIR"
|
||||
export BACKUP_RUN_ID="$run_id" BUILD_SHA SOURCE_IMAGE_DIGEST="$source_image_digest"
|
||||
capture_command paired_backup "$ROOT_DIR/scripts/backup-all.sh"
|
||||
); then
|
||||
fail "paired backup command failed"
|
||||
fi
|
||||
[[ -d "$batch_dir" && -f "$batch_dir/manifest.json" ]] || fail "complete local batch was not published"
|
||||
manifest_sha="$(sha256sum "$batch_dir/manifest.json" | awk '{print $1}')"
|
||||
|
||||
local -a ssh_options=(
|
||||
-i "$OFFHOST_SSH_KEY"
|
||||
-o BatchMode=yes
|
||||
-o IdentitiesOnly=yes
|
||||
-o StrictHostKeyChecking=yes
|
||||
-o "UserKnownHostsFile=$OFFHOST_KNOWN_HOSTS"
|
||||
-o ConnectTimeout=15
|
||||
)
|
||||
local rsync_shell
|
||||
printf -v rsync_shell 'ssh -i %q -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=%q -o ConnectTimeout=15' \
|
||||
"$OFFHOST_SSH_KEY" "$OFFHOST_KNOWN_HOSTS"
|
||||
|
||||
CURRENT_PHASE="offhost_transfer"
|
||||
if ! capture_command offhost_transfer rsync -a --partial --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= -e "$rsync_shell" \
|
||||
"$batch_dir/" "$OFFHOST_TARGET:${batch_name}.partial/"; then
|
||||
fail "off-host transfer failed"
|
||||
fi
|
||||
|
||||
CURRENT_PHASE="offhost_finalize"
|
||||
if ! capture_command offhost_finalize ssh "${ssh_options[@]}" "$OFFHOST_TARGET" \
|
||||
"hyb-backup-finalize $run_id $manifest_sha"; then
|
||||
fail "off-host finalize failed"
|
||||
fi
|
||||
|
||||
CURRENT_PHASE="success_state"
|
||||
write_success_state "$run_id" "$manifest_sha"
|
||||
prune_local_backups "$batch_name"
|
||||
write_summary "PASS_AUTOMATIC_BACKUP_COMPLETE" 0 "offhost_verified"
|
||||
send_summary_alert "automatic-backup-alert-state.json" || true
|
||||
rm -f -- "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log"
|
||||
trap - ERR
|
||||
echo "[automatic-backup][PASS] paired batch finalized off-host"
|
||||
}
|
||||
|
||||
check_stale() {
|
||||
validate_common
|
||||
trap on_error ERR
|
||||
CURRENT_PHASE="stale_check"
|
||||
resolve_active_runtime
|
||||
local success_file="$AUTOMATIC_BACKUP_STATE_DIR/last-success.json"
|
||||
local stale_result
|
||||
if ! stale_result="$(python3 - "$success_file" "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" "$COMPOSE_PROJECT_NAME" "$BUILD_SHA" <<'PYTHON'
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
file_path, max_age, expected_project, expected_build = sys.argv[1:]
|
||||
if not os.path.isfile(file_path):
|
||||
raise SystemExit(1)
|
||||
with open(file_path, "r", encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
if payload.get("schemaVersion") != 1 or payload.get("status") != "complete":
|
||||
raise SystemExit(1)
|
||||
if payload.get("composeProject") != expected_project:
|
||||
raise SystemExit(1)
|
||||
if not re.fullmatch(r"auto-[0-9]{8}T[0-9]{6}Z", payload.get("runId", "")):
|
||||
raise SystemExit(1)
|
||||
if not re.fullmatch(r"[0-9a-f]{64}", payload.get("manifestSha256", "")):
|
||||
raise SystemExit(1)
|
||||
if not re.fullmatch(r"[0-9a-fA-F]{40,64}", payload.get("buildSha", "")):
|
||||
raise SystemExit(1)
|
||||
completed = datetime.fromisoformat(payload["completedAt"].replace("Z", "+00:00"))
|
||||
age = (datetime.now(timezone.utc) - completed).total_seconds()
|
||||
if age < 0 or age > int(max_age):
|
||||
raise SystemExit(1)
|
||||
print("BUILD_MISMATCH" if payload.get("buildSha") != expected_build else "FRESH")
|
||||
PYTHON
|
||||
)"; then
|
||||
fail "latest off-host success is missing, invalid, project-mismatched, or stale"
|
||||
fi
|
||||
if [[ "$stale_result" == "BUILD_MISMATCH" ]]; then
|
||||
write_summary "PASS_WITH_FINDINGS" 1 "recovery_point_build_differs"
|
||||
send_summary_alert "automatic-backup-alert-state.json" || true
|
||||
trap - ERR
|
||||
echo "[automatic-backup][PASS_WITH_FINDINGS] off-host recovery point is fresh but build differs"
|
||||
return 0
|
||||
fi
|
||||
write_summary "PASS_AUTOMATIC_BACKUP_FRESH" 0 "fresh_offhost_recovery_point"
|
||||
send_summary_alert "automatic-backup-alert-state.json" || true
|
||||
trap - ERR
|
||||
echo "[automatic-backup][PASS] off-host recovery point is fresh"
|
||||
}
|
||||
|
||||
remind_drill() {
|
||||
validate_common
|
||||
trap on_error ERR
|
||||
CURRENT_PHASE="quarterly_reminder"
|
||||
resolve_active_runtime
|
||||
local quarter
|
||||
quarter="$(python3 - <<'PYTHON'
|
||||
from datetime import datetime, timezone
|
||||
now = datetime.now(timezone.utc)
|
||||
print(f"{now.year}-Q{((now.month - 1) // 3) + 1}")
|
||||
PYTHON
|
||||
)"
|
||||
write_summary "PASS_WITH_FINDINGS" 1 "quarterly_shadow_restore_due_${quarter}"
|
||||
send_summary_alert "automatic-backup-drill-alert-state.json"
|
||||
trap - ERR
|
||||
echo "[automatic-backup][PASS] quarterly restore reminder evaluated"
|
||||
}
|
||||
|
||||
case "$MODE" in
|
||||
run)
|
||||
run_backup
|
||||
;;
|
||||
check-stale)
|
||||
check_stale
|
||||
;;
|
||||
remind-drill)
|
||||
remind_drill
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: run-automatic-backup.sh [run|check-stale|remind-drill]"
|
||||
;;
|
||||
*)
|
||||
fail "unsupported mode"
|
||||
;;
|
||||
esac
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Send a redacted, deduplicated OA operations alert to a WeCom group robot."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SAFE_TOKEN = re.compile(r"^[A-Za-z0-9_-]{1,160}$")
|
||||
ALLOWED_STATUSES = {
|
||||
"FAIL_AUTOMATIC_BACKUP",
|
||||
"FAIL_OFFHOST_DEADMAN",
|
||||
"PASS_AUTOMATIC_BACKUP_COMPLETE",
|
||||
"PASS_AUTOMATIC_BACKUP_FRESH",
|
||||
"PASS_OFFHOST_DEADMAN_FRESH",
|
||||
"PASS_WITH_FINDINGS",
|
||||
}
|
||||
|
||||
|
||||
def fail(message: str) -> "NoReturn":
|
||||
print(f"[wecom-group-alert][ERROR] {message}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def validate_private_file(path: Path, label: str) -> None:
|
||||
try:
|
||||
info = path.lstat()
|
||||
except FileNotFoundError:
|
||||
fail(f"{label} is missing")
|
||||
if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
|
||||
fail(f"{label} must be a regular non-symbolic file")
|
||||
if stat.S_IMODE(info.st_mode) != 0o600:
|
||||
fail(f"{label} mode must be 0600")
|
||||
if info.st_uid != os.geteuid():
|
||||
fail(f"{label} must be owned by the executing account")
|
||||
|
||||
|
||||
def read_webhook(path: Path) -> str:
|
||||
validate_private_file(path, "webhook file")
|
||||
try:
|
||||
value = path.read_text(encoding="utf-8").strip()
|
||||
except OSError:
|
||||
fail("webhook file cannot be read")
|
||||
if "\n" in value or "\r" in value:
|
||||
fail("webhook file must contain one line")
|
||||
parsed = urllib.parse.urlparse(value)
|
||||
query = urllib.parse.parse_qs(parsed.query, strict_parsing=True)
|
||||
if (
|
||||
parsed.scheme != "https"
|
||||
or parsed.hostname != "qyapi.weixin.qq.com"
|
||||
or parsed.netloc != "qyapi.weixin.qq.com"
|
||||
or parsed.path != "/cgi-bin/webhook/send"
|
||||
or set(query) != {"key"}
|
||||
or len(query["key"]) != 1
|
||||
or not re.fullmatch(r"[A-Za-z0-9-]{16,128}", query["key"][0])
|
||||
or parsed.fragment
|
||||
):
|
||||
fail("webhook file format is invalid")
|
||||
return value
|
||||
|
||||
|
||||
def load_summary(path: Path) -> tuple[str, str, int]:
|
||||
validate_private_file(path, "summary file")
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
fail("summary file is invalid")
|
||||
status = payload.get("status")
|
||||
findings = payload.get("findings") or []
|
||||
finding_count = payload.get("findingCount")
|
||||
if status not in ALLOWED_STATUSES or not isinstance(finding_count, int):
|
||||
fail("summary status is invalid")
|
||||
if finding_count != len(findings) or finding_count not in {0, 1}:
|
||||
fail("summary finding count is invalid")
|
||||
category = "healthy"
|
||||
if findings:
|
||||
category = findings[0].get("detail", "")
|
||||
if not isinstance(category, str) or not SAFE_TOKEN.fullmatch(category):
|
||||
fail("summary category is invalid")
|
||||
return status, category, finding_count
|
||||
|
||||
|
||||
def load_previous(path: Path) -> dict[str, object]:
|
||||
if not path.exists():
|
||||
return {}
|
||||
validate_private_file(path, "alert state file")
|
||||
try:
|
||||
value = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
fail("alert state file is invalid")
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def atomic_write(path: Path, payload: dict[str, object]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
os.chmod(path.parent, 0o700)
|
||||
fd, temp_name = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
|
||||
try:
|
||||
os.fchmod(fd, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, ensure_ascii=True, indent=2)
|
||||
handle.write("\n")
|
||||
os.replace(temp_name, path)
|
||||
except BaseException:
|
||||
try:
|
||||
os.unlink(temp_name)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def should_send(previous: dict[str, object], status: str, category: str) -> bool:
|
||||
previous_status = previous.get("status")
|
||||
previous_category = previous.get("category")
|
||||
if status.startswith("FAIL_") or status == "PASS_WITH_FINDINGS":
|
||||
return previous_status != status or previous_category != category
|
||||
return isinstance(previous_status, str) and (
|
||||
previous_status.startswith("FAIL_") or previous_status == "PASS_WITH_FINDINGS"
|
||||
)
|
||||
|
||||
|
||||
def safe_text(status: str, category: str) -> str:
|
||||
if status.startswith("FAIL_"):
|
||||
headline = "内部 OA 运维告警"
|
||||
elif status == "PASS_WITH_FINDINGS":
|
||||
headline = "内部 OA 运维提醒"
|
||||
else:
|
||||
headline = "内部 OA 运维恢复通知"
|
||||
return f"{headline}\n状态：{status}\n类别：{category}\n请按恢复卡检查，不要在群内粘贴原始日志。"
|
||||
|
||||
|
||||
def send(webhook: str, text: str) -> None:
|
||||
body = json.dumps({"msgtype": "text", "text": {"content": text}}).encode("utf-8")
|
||||
capture = os.environ.get("HYB_WECOM_ALERT_TEST_CAPTURE")
|
||||
if capture:
|
||||
if os.environ.get("HYB_WECOM_ALERT_TEST_MODE") != "1":
|
||||
fail("test capture is disabled")
|
||||
capture_path = Path(capture)
|
||||
with capture_path.open("ab") as handle:
|
||||
handle.write(body + b"\n")
|
||||
os.chmod(capture_path, 0o600)
|
||||
return
|
||||
request = urllib.request.Request(
|
||||
webhook,
|
||||
data=body,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
result = json.loads(response.read().decode("utf-8"))
|
||||
except Exception:
|
||||
fail("group robot request failed")
|
||||
if result.get("errcode") != 0:
|
||||
fail("group robot rejected the alert")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--webhook-file", required=True, type=Path)
|
||||
parser.add_argument("--summary-file", required=True, type=Path)
|
||||
parser.add_argument("--state-file", required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
status, category, _ = load_summary(args.summary_file)
|
||||
previous = load_previous(args.state_file)
|
||||
send_now = should_send(previous, status, category)
|
||||
if send_now:
|
||||
webhook = read_webhook(args.webhook_file)
|
||||
send(webhook, safe_text(status, category))
|
||||
atomic_write(args.state_file, {"schemaVersion": 1, "status": status, "category": category})
|
||||
except OSError:
|
||||
fail("local alert file operation failed")
|
||||
print(f"[wecom-group-alert][PASS] evaluated sent={str(send_now).lower()}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -29,6 +29,7 @@
|
||||
"audit:smoke-data": "node src/scripts/audit-smoke-data.js",
|
||||
"audit:org-master-data": "node src/scripts/audit-org-master-data.js",
|
||||
"audit:wecom-oa-users": "node src/scripts/audit-wecom-oa-user-consistency.js",
|
||||
"import:wecom-directory-export": "node src/scripts/import-wecom-directory-export.js",
|
||||
"repair:oa-only-smoke": "node src/scripts/repair-oa-only-smoke-accounts.js",
|
||||
"check:wecom-approval-fixtures": "node src/scripts/check-wecom-approval-fixtures.js",
|
||||
"check:wecom-approval-callback-poc": "node src/scripts/check-wecom-approval-callback-poc.js",
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
-- Enterprise WeChat directory exports may omit a member phone number.
|
||||
-- PostgreSQL unique constraints continue to reject duplicate non-null phones
|
||||
-- while allowing multiple employees to remain temporarily unfilled.
|
||||
ALTER TABLE "employees"
|
||||
ALTER COLUMN "phone" DROP NOT NULL;
|
||||
@@ -578,7 +578,7 @@ model Employee {
|
||||
id String @id @default(uuid()) @db.Uuid
|
||||
employee_no String @unique @default(dbgenerated("generate_employee_number()"))
|
||||
name String
|
||||
phone String @unique
|
||||
phone String? @unique
|
||||
email String @unique
|
||||
department_id String @db.Uuid
|
||||
legal_entity_id String? @db.Uuid
|
||||
|
||||
+19
-14
@@ -58,10 +58,15 @@ const normalizedEmailSchema = z
|
||||
.max(200)
|
||||
.transform((value) => value.toLowerCase());
|
||||
|
||||
const nullablePhoneSchema = z.preprocess(
|
||||
(value) => (typeof value === "string" && value.trim() === "" ? null : value),
|
||||
z.string().trim().min(6).max(30).nullable().optional(),
|
||||
);
|
||||
|
||||
const createBodySchema = z.object({
|
||||
employeeNo: z.string().trim().min(1).max(50).optional(),
|
||||
name: z.string().min(1).max(100),
|
||||
phone: z.string().min(6).max(30),
|
||||
phone: nullablePhoneSchema,
|
||||
email: normalizedEmailSchema,
|
||||
departmentId: z.string().uuid().optional(),
|
||||
title: z.string().min(1).max(100).optional(),
|
||||
@@ -74,7 +79,7 @@ const createBodySchema = z.object({
|
||||
|
||||
const updateBodySchema = z.object({
|
||||
name: z.string().trim().min(1).max(100).optional(),
|
||||
phone: z.string().trim().min(6).max(30).optional(),
|
||||
phone: nullablePhoneSchema,
|
||||
email: normalizedEmailSchema.optional(),
|
||||
departmentId: z.string().uuid().optional(),
|
||||
title: z.string().trim().min(1).max(100).optional(),
|
||||
@@ -98,8 +103,8 @@ const sensitiveBodySchema = z
|
||||
})
|
||||
.strict();
|
||||
|
||||
const USER_IMPORT_REQUIRED_HEADERS = ["employeeNo", "name", "phone", "email", "departmentName"];
|
||||
const USER_IMPORT_OPTIONAL_HEADERS = ["title", "directManagerEmployeeNo", "roles", "hireDate"];
|
||||
const USER_IMPORT_REQUIRED_HEADERS = ["employeeNo", "name", "email", "departmentName"];
|
||||
const USER_IMPORT_OPTIONAL_HEADERS = ["phone", "title", "directManagerEmployeeNo", "roles", "hireDate"];
|
||||
const USER_IMPORT_ALLOWED_EXTENSIONS = new Set([".csv", ".xlsx"]);
|
||||
const USER_IMPORT_ALLOWED_MIME_TYPES = new Set([
|
||||
"text/csv",
|
||||
@@ -773,11 +778,9 @@ function validateImportRowFields(row) {
|
||||
pushImportRowError(row, "name", "too_long");
|
||||
}
|
||||
|
||||
if (!normalized.phone) {
|
||||
pushImportRowError(row, "phone", "required");
|
||||
} else if (normalized.phone.length < 6) {
|
||||
if (normalized.phone && normalized.phone.length < 6) {
|
||||
pushImportRowError(row, "phone", "too_short");
|
||||
} else if (normalized.phone.length > 30) {
|
||||
} else if (normalized.phone && normalized.phone.length > 30) {
|
||||
pushImportRowError(row, "phone", "too_long");
|
||||
}
|
||||
|
||||
@@ -834,7 +837,7 @@ function createDepartmentResolutionMap(departments) {
|
||||
function createEmployeeLookupMaps(employees) {
|
||||
return {
|
||||
byEmployeeNo: new Map(employees.map((item) => [item.employee_no, item])),
|
||||
byPhone: new Map(employees.map((item) => [item.phone, item])),
|
||||
byPhone: new Map(employees.filter((item) => item.phone).map((item) => [item.phone, item])),
|
||||
byEmail: new Map(employees.map((item) => [item.email, item])),
|
||||
};
|
||||
}
|
||||
@@ -912,7 +915,9 @@ async function buildUserImportPreview(prisma, rawRows, options = {}) {
|
||||
pushImportRowError(row, "employeeNo", "duplicate_existing");
|
||||
}
|
||||
|
||||
const phoneConflict = existingLookup.byPhone.get(row.normalized.phone);
|
||||
const phoneConflict = row.normalized.phone
|
||||
? existingLookup.byPhone.get(row.normalized.phone)
|
||||
: null;
|
||||
if (phoneConflict) {
|
||||
pushImportRowError(row, "phone", "duplicate_existing");
|
||||
}
|
||||
@@ -1012,7 +1017,7 @@ async function createSystemUser(tx, options) {
|
||||
? { employee_no: options.employeeNo.trim() }
|
||||
: {}),
|
||||
name: options.name,
|
||||
phone: options.phone,
|
||||
phone: options.phone || null,
|
||||
email: options.email.trim().toLowerCase(),
|
||||
department_id: options.departmentId,
|
||||
title: options.title ?? "员工",
|
||||
@@ -1573,7 +1578,7 @@ async function usersRoutes(fastify, opts = {}) {
|
||||
const employee = await createSystemUser(tx, {
|
||||
employeeNo: row.normalized.employeeNo,
|
||||
name: row.normalized.name,
|
||||
phone: row.normalized.phone,
|
||||
phone: row.normalized.phone || null,
|
||||
email: row.normalized.email,
|
||||
departmentId: row.resolvedDepartmentId,
|
||||
title: row.normalized.title,
|
||||
@@ -1742,7 +1747,7 @@ async function usersRoutes(fastify, opts = {}) {
|
||||
if (typeof body.name === "string" && body.name !== current.name) {
|
||||
changes.name = { before: current.name, after: body.name };
|
||||
}
|
||||
if (typeof body.phone === "string" && body.phone !== current.phone) {
|
||||
if (Object.prototype.hasOwnProperty.call(body, "phone") && (body.phone ?? null) !== current.phone) {
|
||||
changes.phone = { before: current.phone, after: body.phone };
|
||||
}
|
||||
if (typeof body.email === "string" && body.email !== current.email) {
|
||||
@@ -1768,7 +1773,7 @@ async function usersRoutes(fastify, opts = {}) {
|
||||
where: { id: userId },
|
||||
data: {
|
||||
...(typeof body.name === "string" ? { name: body.name } : {}),
|
||||
...(typeof body.phone === "string" ? { phone: body.phone } : {}),
|
||||
...(Object.prototype.hasOwnProperty.call(body, "phone") ? { phone: body.phone ?? null } : {}),
|
||||
...(typeof body.email === "string" ? { email: body.email } : {}),
|
||||
...(typeof body.departmentId === "string" ? { department_id: body.departmentId } : {}),
|
||||
...(typeof body.title === "string" ? { title: body.title } : {}),
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
#!/usr/bin/env node
|
||||
require("dotenv/config");
|
||||
|
||||
const fs = require("node:fs");
|
||||
const ExcelJS = require("exceljs");
|
||||
|
||||
const { createPrismaClient } = require("../prismaClient");
|
||||
const {
|
||||
maskEmail,
|
||||
maskExternalUserId,
|
||||
maskPhone,
|
||||
runWecomHrMasterSync,
|
||||
} = require("../services/integrations/wecomHrMaster");
|
||||
|
||||
const APPLY_CONFIRM_ENV = "WECOM_DIRECTORY_EXPORT_APPLY";
|
||||
const REQUIRED_HEADERS = ["姓名", "账号", "部门", "企业邮箱"];
|
||||
const SAFE_EMPTY_PHONE = null;
|
||||
|
||||
class DirectoryExportError extends Error {
|
||||
constructor(code, details = {}) {
|
||||
super(code);
|
||||
this.name = "DirectoryExportError";
|
||||
this.code = code;
|
||||
this.details = details;
|
||||
}
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
process.stdout.write(`Usage:
|
||||
node src/scripts/import-wecom-directory-export.js [--input=-] [--dry-run] [--format=json]
|
||||
WECOM_DIRECTORY_EXPORT_APPLY=1 node src/scripts/import-wecom-directory-export.js --input=- --apply --actor-employee-no=<no>
|
||||
|
||||
Reads an Enterprise WeChat directory XLSX from stdin. Dry-run is the default.
|
||||
Reports contain counts and categories only; personal field values are never printed.
|
||||
`);
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const options = {
|
||||
input: "-",
|
||||
mode: "dry_run",
|
||||
format: "text",
|
||||
actorEmployeeNo: "",
|
||||
help: false,
|
||||
};
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index];
|
||||
if (arg === "--help" || arg === "-h") options.help = true;
|
||||
else if (arg === "--dry-run") options.mode = "dry_run";
|
||||
else if (arg === "--apply") options.mode = "apply";
|
||||
else if (arg === "--input") options.input = argv[++index] ?? "";
|
||||
else if (arg.startsWith("--input=")) options.input = arg.slice("--input=".length);
|
||||
else if (arg === "--format") options.format = argv[++index] ?? "";
|
||||
else if (arg.startsWith("--format=")) options.format = arg.slice("--format=".length);
|
||||
else if (arg === "--actor-employee-no") options.actorEmployeeNo = argv[++index] ?? "";
|
||||
else if (arg.startsWith("--actor-employee-no=")) {
|
||||
options.actorEmployeeNo = arg.slice("--actor-employee-no=".length);
|
||||
} else throw new DirectoryExportError("unsupported_argument");
|
||||
}
|
||||
if (options.input !== "-") throw new DirectoryExportError("stdin_required");
|
||||
if (!new Set(["text", "json"]).has(options.format)) throw new DirectoryExportError("invalid_format");
|
||||
if (options.mode === "apply") {
|
||||
if (process.env[APPLY_CONFIRM_ENV] !== "1") throw new DirectoryExportError("apply_confirmation_required");
|
||||
if (!/^[A-Za-z0-9_-]{1,80}$/.test(options.actorEmployeeNo)) {
|
||||
throw new DirectoryExportError("actor_required");
|
||||
}
|
||||
}
|
||||
return options;
|
||||
}
|
||||
|
||||
function cellToString(value) {
|
||||
if (value == null) return "";
|
||||
if (value instanceof Date) return value.toISOString().slice(0, 10);
|
||||
if (typeof value === "object") {
|
||||
if (Array.isArray(value.richText)) return value.richText.map((item) => item.text ?? "").join("").trim();
|
||||
if (value.text != null) return String(value.text).trim();
|
||||
if (value.result != null) return cellToString(value.result);
|
||||
}
|
||||
return String(value).trim();
|
||||
}
|
||||
|
||||
function findHeaderRow(worksheet) {
|
||||
const maximum = Math.min(50, worksheet.rowCount);
|
||||
for (let rowNumber = 1; rowNumber <= maximum; rowNumber += 1) {
|
||||
const labels = worksheet.getRow(rowNumber).values.slice(1).map(cellToString);
|
||||
if (REQUIRED_HEADERS.every((header) => labels.includes(header))) {
|
||||
return { rowNumber, labels };
|
||||
}
|
||||
}
|
||||
throw new DirectoryExportError("required_headers_missing", { missingHeaderCount: REQUIRED_HEADERS.length });
|
||||
}
|
||||
|
||||
function normalizeEmail(value) {
|
||||
const raw = cellToString(value).toLowerCase();
|
||||
return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(raw) ? raw : "";
|
||||
}
|
||||
|
||||
function normalizePhone(value) {
|
||||
const raw = cellToString(value);
|
||||
if (!raw) return { value: SAFE_EMPTY_PHONE, invalid: false };
|
||||
const normalized = raw.replace(/[\s-]/g, "");
|
||||
if (!/^\+?\d{6,30}$/.test(normalized)) {
|
||||
return { value: SAFE_EMPTY_PHONE, invalid: true };
|
||||
}
|
||||
return { value: normalized, invalid: false };
|
||||
}
|
||||
|
||||
function buildProfile(data) {
|
||||
const externalUserId = cellToString(data["账号"]).slice(0, 128);
|
||||
const name = cellToString(data["姓名"]).slice(0, 100);
|
||||
const phone = normalizePhone(data["手机"]);
|
||||
const email = normalizeEmail(data["企业邮箱"]);
|
||||
const departmentName = cellToString(data["部门"]).slice(0, 100);
|
||||
const title = cellToString(data["职务"]).slice(0, 100) || "员工";
|
||||
const activation = cellToString(data["激活状态"]);
|
||||
const disabled = cellToString(data["禁用状态"]);
|
||||
const active = (!activation || activation === "已激活")
|
||||
&& (!disabled || new Set(["否", "未禁用", "正常", "0"]).has(disabled));
|
||||
const missingRequiredFields = [];
|
||||
for (const [field, value] of [
|
||||
["externalUserId", externalUserId],
|
||||
["name", name],
|
||||
["email", email],
|
||||
["department", departmentName],
|
||||
]) {
|
||||
if (!value) missingRequiredFields.push(field);
|
||||
}
|
||||
if (phone.invalid) missingRequiredFields.push("phone_invalid");
|
||||
return {
|
||||
externalUserId,
|
||||
name,
|
||||
phone: phone.value,
|
||||
email,
|
||||
title,
|
||||
departmentExternalId: departmentName,
|
||||
departmentName,
|
||||
managerExternalUserId: "",
|
||||
hireDate: null,
|
||||
sourceStatus: active ? "active" : "inactive",
|
||||
missingRequiredFields,
|
||||
sourceSummary: {
|
||||
externalUserIdPrefix: maskExternalUserId(externalUserId),
|
||||
phoneMasked: maskPhone(phone.value),
|
||||
emailMasked: maskEmail(email),
|
||||
hasPhone: Boolean(phone.value),
|
||||
hasEmail: Boolean(email),
|
||||
hasDepartment: Boolean(departmentName),
|
||||
sourceStatus: active ? "active" : "inactive",
|
||||
sourceKind: "wecom_directory_export",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function countDuplicateValues(profiles, key) {
|
||||
const counts = new Map();
|
||||
for (const profile of profiles) {
|
||||
const value = String(profile[key] ?? "").trim().toLowerCase();
|
||||
if (!value) continue;
|
||||
counts.set(value, (counts.get(value) ?? 0) + 1);
|
||||
}
|
||||
return [...counts.values()].filter((count) => count > 1).length;
|
||||
}
|
||||
|
||||
async function parseDirectoryExport(buffer) {
|
||||
const workbook = new ExcelJS.Workbook();
|
||||
await workbook.xlsx.load(buffer);
|
||||
const worksheet = workbook.worksheets[0];
|
||||
if (!worksheet) throw new DirectoryExportError("worksheet_missing");
|
||||
const header = findHeaderRow(worksheet);
|
||||
const profiles = [];
|
||||
worksheet.eachRow((row, rowNumber) => {
|
||||
if (rowNumber <= header.rowNumber) return;
|
||||
const data = {};
|
||||
header.labels.forEach((label, index) => {
|
||||
if (label) data[label] = cellToString(row.getCell(index + 1).value);
|
||||
});
|
||||
if (Object.values(data).some(Boolean)) profiles.push(buildProfile(data));
|
||||
});
|
||||
if (!profiles.length) throw new DirectoryExportError("member_rows_missing");
|
||||
|
||||
const duplicateCounts = {
|
||||
account: countDuplicateValues(profiles, "externalUserId"),
|
||||
phone: countDuplicateValues(profiles, "phone"),
|
||||
email: countDuplicateValues(profiles, "email"),
|
||||
};
|
||||
if (Object.values(duplicateCounts).some((count) => count > 0)) {
|
||||
throw new DirectoryExportError("duplicate_values", duplicateCounts);
|
||||
}
|
||||
return {
|
||||
profiles,
|
||||
sourceSummary: {
|
||||
memberCount: profiles.length,
|
||||
activeCount: profiles.filter((profile) => profile.sourceStatus === "active").length,
|
||||
phonePresentCount: profiles.filter((profile) => Boolean(profile.phone)).length,
|
||||
phoneMissingCount: profiles.filter((profile) => !profile.phone).length,
|
||||
emailPresentCount: profiles.filter((profile) => Boolean(profile.email)).length,
|
||||
incompleteCount: profiles.filter((profile) => profile.missingRequiredFields.length > 0).length,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function resolveActorId(prisma, employeeNo) {
|
||||
if (!employeeNo) return null;
|
||||
const employee = await prisma.employee.findUnique({
|
||||
where: { employee_no: employeeNo },
|
||||
select: { id: true, status: true, archived_at: true },
|
||||
});
|
||||
if (!employee || employee.status !== "active" || employee.archived_at) {
|
||||
throw new DirectoryExportError("actor_unavailable");
|
||||
}
|
||||
return employee.id;
|
||||
}
|
||||
|
||||
function buildSafeReport(options, parsed, syncRun) {
|
||||
const summary = syncRun?.summary ?? {};
|
||||
const blocked = Number(summary.failed ?? 0) > 0
|
||||
|| Number(summary.conflicts ?? 0) > 0
|
||||
|| Number(summary.skipped ?? 0) > 0
|
||||
|| parsed.sourceSummary.incompleteCount > 0;
|
||||
return {
|
||||
issueId: "WECOM_DIRECTORY_EXPORT",
|
||||
status: blocked
|
||||
? "BLOCKED_DIRECTORY_EXPORT"
|
||||
: options.mode === "apply"
|
||||
? "PASS_DIRECTORY_EXPORT_APPLY"
|
||||
: "PASS_DIRECTORY_EXPORT_DRY_RUN",
|
||||
mode: options.mode,
|
||||
source: parsed.sourceSummary,
|
||||
sync: {
|
||||
created: Number(summary.created ?? 0),
|
||||
updated: Number(summary.updated ?? 0),
|
||||
noop: Number(summary.noop ?? 0),
|
||||
skipped: Number(summary.skipped ?? 0),
|
||||
conflicts: Number(summary.conflicts ?? 0),
|
||||
failed: Number(summary.failed ?? 0),
|
||||
planned: Number(summary.planned ?? 0),
|
||||
success: Number(summary.success ?? 0),
|
||||
},
|
||||
privacy: "counts/categories only; no names, accounts, phones, emails, departments, paths, or raw rows",
|
||||
};
|
||||
}
|
||||
|
||||
function printReport(report, format) {
|
||||
if (format === "json") {
|
||||
process.stdout.write(`${JSON.stringify(report)}\n`);
|
||||
return;
|
||||
}
|
||||
process.stdout.write(`[wecom-directory-export] status=${report.status} mode=${report.mode}\n`);
|
||||
process.stdout.write(`[wecom-directory-export] members=${report.source.memberCount} phonePresent=${report.source.phonePresentCount} phoneMissing=${report.source.phoneMissingCount} incomplete=${report.source.incompleteCount}\n`);
|
||||
process.stdout.write(`[wecom-directory-export] planned=${report.sync.planned} conflicts=${report.sync.conflicts} skipped=${report.sync.skipped} failed=${report.sync.failed}\n`);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
let resources = null;
|
||||
try {
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
if (options.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
const parsed = await parseDirectoryExport(await fs.promises.readFile(0));
|
||||
resources = createPrismaClient();
|
||||
const actorId = await resolveActorId(resources.prisma, options.actorEmployeeNo);
|
||||
const syncRun = await runWecomHrMasterSync({
|
||||
prisma: resources.prisma,
|
||||
config: null,
|
||||
actorId,
|
||||
mode: options.mode,
|
||||
limit: parsed.profiles.length,
|
||||
sourceProfiles: parsed.profiles,
|
||||
sourceLabel: "manual_export",
|
||||
});
|
||||
const report = buildSafeReport(options, parsed, syncRun);
|
||||
printReport(report, options.format);
|
||||
if (report.status.startsWith("BLOCKED")) process.exitCode = 2;
|
||||
} catch (error) {
|
||||
const code = error instanceof DirectoryExportError ? error.code : "directory_export_failed";
|
||||
process.stderr.write(`[wecom-directory-export][ERROR] ${code}\n`);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
await resources?.prisma?.$disconnect?.().catch(() => {});
|
||||
await resources?.pool?.end?.().catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
if (require.main === module) main();
|
||||
|
||||
module.exports = {
|
||||
DirectoryExportError,
|
||||
buildProfile,
|
||||
buildSafeReport,
|
||||
countDuplicateValues,
|
||||
findHeaderRow,
|
||||
parseArgs,
|
||||
parseDirectoryExport,
|
||||
};
|
||||
@@ -16,11 +16,11 @@ const SEND_CONFIRM_ENV = "WARCH3_OPS_ALERT_SEND";
|
||||
function printHelp() {
|
||||
process.stdout.write(`Usage: node src/scripts/send-wecom-ops-alert.js [options]
|
||||
|
||||
QA-only WARCH3 企微巡检脱敏告警。
|
||||
QA/运维脱敏告警，支持 WARCH3 巡检和 AUTO_BACKUP 自动备份状态。
|
||||
默认 dry-run：只生成预览和报告，不发送企微消息。
|
||||
|
||||
Options:
|
||||
--summary-file=<path> WARCH3/WROAD2 summary.json。
|
||||
--summary-file=<path|-> WARCH3/WROAD2 summary.json；- 表示从 stdin 读取。
|
||||
--recipient-employee-no=<no> 收件员工号，可重复；默认 HYD000006。
|
||||
--send 发送企微应用消息。
|
||||
--force PASS 且无 finding 时也发送测试消息。
|
||||
@@ -131,7 +131,12 @@ function assertAlertSendAllowed(options, env = process.env) {
|
||||
}
|
||||
}
|
||||
|
||||
function readSummary(summaryFile) {
|
||||
function safeIssueId(value) {
|
||||
const issueId = toSafeString(value, 40);
|
||||
return /^[A-Z0-9_-]{1,40}$/.test(issueId) ? issueId : "WARCH3";
|
||||
}
|
||||
|
||||
function readSummary(summaryFile, stdinText) {
|
||||
if (!summaryFile) {
|
||||
return {
|
||||
status: "PARTIAL_ALERT_NEEDS_CONFIG",
|
||||
@@ -141,12 +146,17 @@ function readSummary(summaryFile) {
|
||||
notes: "summary_file_missing",
|
||||
};
|
||||
}
|
||||
const resolved = path.resolve(summaryFile);
|
||||
const raw = fs.readFileSync(resolved, "utf8");
|
||||
const fromStdin = summaryFile === "-";
|
||||
const resolved = fromStdin ? null : path.resolve(summaryFile);
|
||||
const raw = fromStdin
|
||||
? (stdinText ?? fs.readFileSync(0, "utf8"))
|
||||
: fs.readFileSync(resolved, "utf8");
|
||||
const parsed = JSON.parse(raw);
|
||||
return {
|
||||
...parsed,
|
||||
reportDir: parsed.reportDir || path.dirname(resolved),
|
||||
issueId: safeIssueId(parsed.issueId),
|
||||
// stdin is an untrusted cross-host boundary. Never echo a supplied path.
|
||||
reportDir: fromStdin ? "stdin-summary" : (parsed.reportDir || path.dirname(resolved)),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -156,7 +166,20 @@ function shouldSendForSummary(summary, options) {
|
||||
}
|
||||
|
||||
function nextStepForStatus(summary) {
|
||||
const issueId = safeIssueId(summary?.issueId);
|
||||
const status = toSafeString(summary?.status || summary?.conclusion, 80);
|
||||
const findingText = Array.isArray(summary?.findings)
|
||||
? summary.findings.map((finding) => toSafeString(finding?.detail, 160)).join(" ")
|
||||
: "";
|
||||
if (issueId === "AUTO_BACKUP" && /quarterly_shadow_restore_due/i.test(findingText)) {
|
||||
return "按恢复卡在 ARM 隔离目标执行人工影子恢复，并核对就绪状态与关键计数。";
|
||||
}
|
||||
if (issueId === "AUTO_BACKUP" && (
|
||||
/FAIL|STALE|MISSING|INVALID|MISMATCH|BLOCKED|ERROR/i.test(status)
|
||||
|| Number(summary?.findingCount || 0) > 0
|
||||
)) {
|
||||
return "检查自动备份任务、异机传输链路和最近一个已校验成功的恢复点。";
|
||||
}
|
||||
if (status === "SKIP/BLOCKED_BY_EXTERNAL_CONFIG") {
|
||||
return "检查 Secret、权限、可信 IP、可见范围或外部配置后复跑。";
|
||||
}
|
||||
@@ -167,13 +190,13 @@ function nextStepForStatus(summary) {
|
||||
}
|
||||
|
||||
function buildAlertMessage(summary) {
|
||||
const issueId = toSafeString(summary?.issueId || "WARCH3", 40) || "WARCH3";
|
||||
const issueId = safeIssueId(summary?.issueId);
|
||||
const status = toSafeString(summary?.status || summary?.conclusion || "UNKNOWN", 80) || "UNKNOWN";
|
||||
const findingCount = Number(summary?.findingCount || 0);
|
||||
const reportPath = toSafeString(summary?.reportDir || summary?.sourceReports?.wroad2 || "N/A", 500);
|
||||
const nextStep = nextStepForStatus(summary);
|
||||
return {
|
||||
title: `OA 巡检 ${status}`,
|
||||
title: issueId === "AUTO_BACKUP" ? `OA 自动备份 ${status}` : `OA 巡检 ${status}`,
|
||||
body: [
|
||||
`阶段：${issueId}`,
|
||||
`状态：${status}`,
|
||||
@@ -252,6 +275,7 @@ function writeAlertState(stateFile, state) {
|
||||
|
||||
function evaluateAlertDecision(summary, options = {}, priorState = null) {
|
||||
const currentState = buildAlertState(summary);
|
||||
const issueId = safeIssueId(summary?.issueId);
|
||||
if (options.force) {
|
||||
return {
|
||||
shouldSend: true,
|
||||
@@ -267,8 +291,31 @@ function evaluateAlertDecision(summary, options = {}, priorState = null) {
|
||||
const previousRank = Number(priorState?.statusRank || statusRank(priorState?.status));
|
||||
const previousKeys = new Set(Array.isArray(priorState?.findingKeys) ? priorState.findingKeys : []);
|
||||
const newFindingKeys = currentState.findingKeys.filter((key) => !previousKeys.has(key));
|
||||
const isAutomaticBackup = issueId === "AUTO_BACKUP";
|
||||
const isQuarterlyReminder = currentState.findingKeys.some((key) => /quarterly_shadow_restore_due/i.test(key));
|
||||
|
||||
if (isAutomaticBackup && isQuarterlyReminder && (!priorState || newFindingKeys.length > 0)) {
|
||||
return {
|
||||
shouldSend: true,
|
||||
reason: "quarterly_restore_reminder",
|
||||
newFindingKeys,
|
||||
previousStatus: priorState?.status || null,
|
||||
currentStatus: currentState.status,
|
||||
currentState,
|
||||
};
|
||||
}
|
||||
|
||||
if (currentRank >= 4) {
|
||||
if (isAutomaticBackup && priorState && currentRank <= previousRank && newFindingKeys.length === 0) {
|
||||
return {
|
||||
shouldSend: false,
|
||||
reason: "repeated_automatic_backup_failure",
|
||||
newFindingKeys,
|
||||
previousStatus: priorState.status || null,
|
||||
currentStatus: currentState.status,
|
||||
currentState,
|
||||
};
|
||||
}
|
||||
return {
|
||||
shouldSend: true,
|
||||
reason: "blocked_or_failed",
|
||||
@@ -412,6 +459,7 @@ async function executeOpsAlert(options, deps = {}) {
|
||||
const generatedAt = deps.generatedAt || new Date().toISOString();
|
||||
const config = deps.config || getConfig();
|
||||
const summary = deps.summary || readSummary(options.summaryFile);
|
||||
const issueId = safeIssueId(summary?.issueId);
|
||||
const message = buildAlertMessage(summary);
|
||||
const priorState = Object.prototype.hasOwnProperty.call(deps, "priorState")
|
||||
? deps.priorState
|
||||
@@ -497,7 +545,7 @@ async function executeOpsAlert(options, deps = {}) {
|
||||
}
|
||||
|
||||
return {
|
||||
issueId: "WARCH3",
|
||||
issueId,
|
||||
mode: "wecom-ops-alert",
|
||||
generatedAt,
|
||||
status,
|
||||
@@ -513,7 +561,7 @@ async function executeOpsAlert(options, deps = {}) {
|
||||
},
|
||||
stateWrite,
|
||||
summary: {
|
||||
issueId: summary?.issueId || null,
|
||||
issueId,
|
||||
status: summary?.status || summary?.conclusion || "UNKNOWN",
|
||||
findingCount: Number(summary?.findingCount || 0),
|
||||
reportDir: summary?.reportDir || null,
|
||||
@@ -572,6 +620,7 @@ module.exports = {
|
||||
evaluateAlertDecision,
|
||||
executeOpsAlert,
|
||||
parseArgs,
|
||||
readSummary,
|
||||
readAlertState,
|
||||
sanitizeDeliveryResponse,
|
||||
shouldSendForSummary,
|
||||
|
||||
@@ -468,7 +468,7 @@ function normalizeWecomHrProfile({ contact = {}, staff = {}, departmentMap = new
|
||||
};
|
||||
|
||||
profile.missingRequiredFields = [];
|
||||
for (const field of ["externalUserId", "name", "phone", "email"]) {
|
||||
for (const field of ["externalUserId", "name", "email"]) {
|
||||
if (!isMeaningfulString(profile[field])) profile.missingRequiredFields.push(field);
|
||||
}
|
||||
if (!isMeaningfulString(profile.departmentExternalId) && !isMeaningfulString(profile.departmentName)) {
|
||||
@@ -721,18 +721,21 @@ async function detectUniqueConflicts(prisma, employeeId, profile) {
|
||||
|
||||
function buildEmployeeChanges(employee, profile, department) {
|
||||
if (!employee) {
|
||||
return {
|
||||
const changes = {
|
||||
name: buildChange("name", null, profile.name),
|
||||
phone: buildChange("phone", null, profile.phone),
|
||||
email: buildChange("email", null, profile.email),
|
||||
department: buildChange("department", null, department.name),
|
||||
title: buildChange("title", null, profile.title),
|
||||
};
|
||||
if (profile.phone) changes.phone = buildChange("phone", null, profile.phone);
|
||||
return changes;
|
||||
}
|
||||
|
||||
const changes = {};
|
||||
if (employee.name !== profile.name) changes.name = buildChange("name", employee.name, profile.name);
|
||||
if (employee.phone !== profile.phone) changes.phone = buildChange("phone", employee.phone, profile.phone);
|
||||
if (profile.phone && employee.phone !== profile.phone) {
|
||||
changes.phone = buildChange("phone", employee.phone, profile.phone);
|
||||
}
|
||||
if (employee.email !== profile.email) changes.email = buildChange("email", employee.email, profile.email);
|
||||
if (department.id && employee.department_id !== department.id) {
|
||||
changes.department = buildChange("department", employee.department_id, department.name);
|
||||
@@ -854,7 +857,7 @@ async function processProfile(prisma, runId, profile, context) {
|
||||
where: { id: existing.id },
|
||||
data: {
|
||||
name: profile.name,
|
||||
phone: profile.phone,
|
||||
...(profile.phone ? { phone: profile.phone } : {}),
|
||||
email: profile.email,
|
||||
department_id: department.id,
|
||||
title: profile.title,
|
||||
@@ -866,7 +869,7 @@ async function processProfile(prisma, runId, profile, context) {
|
||||
: await prisma.employee.create({
|
||||
data: {
|
||||
name: profile.name,
|
||||
phone: profile.phone,
|
||||
phone: profile.phone || null,
|
||||
email: profile.email,
|
||||
department_id: department.id,
|
||||
title: profile.title,
|
||||
@@ -1077,6 +1080,8 @@ async function runWecomHrMasterSync({
|
||||
userids: rawUserids,
|
||||
limit: rawLimit,
|
||||
fetchImpl = fetch,
|
||||
sourceProfiles = null,
|
||||
sourceLabel = "manual_export",
|
||||
}) {
|
||||
const mode = normalizeMode(rawMode);
|
||||
const userids = normalizeUserids(rawUserids);
|
||||
@@ -1095,12 +1100,17 @@ async function runWecomHrMasterSync({
|
||||
|
||||
let source = null;
|
||||
try {
|
||||
source = await fetchWecomHrProfiles({
|
||||
config,
|
||||
userids,
|
||||
limit,
|
||||
fetchImpl,
|
||||
});
|
||||
source = Array.isArray(sourceProfiles)
|
||||
? {
|
||||
profiles: sourceProfiles.slice(0, limit),
|
||||
tokenSource: { offline: toSafeString(sourceLabel, 40) || "manual_export" },
|
||||
}
|
||||
: await fetchWecomHrProfiles({
|
||||
config,
|
||||
userids,
|
||||
limit,
|
||||
fetchImpl,
|
||||
});
|
||||
} catch (error) {
|
||||
const status = error instanceof WecomHrMasterSyncError && error.code === "wecom_token_unavailable"
|
||||
? "skipped"
|
||||
|
||||
@@ -115,7 +115,7 @@ async function fetchWecomOAuthOnboardingProfile({
|
||||
};
|
||||
|
||||
const missing = [];
|
||||
for (const field of ["name", "phone", "email", "departmentExternalId", "departmentName"]) {
|
||||
for (const field of ["name", "email", "departmentExternalId", "departmentName"]) {
|
||||
if (!profile[field]) missing.push(field);
|
||||
}
|
||||
if (missing.length) {
|
||||
@@ -186,7 +186,12 @@ async function resolveEmployee(tx, profile) {
|
||||
},
|
||||
});
|
||||
const contactMatches = await tx.employee.findMany({
|
||||
where: { OR: [{ phone: profile.phone }, { email: profile.email }] },
|
||||
where: {
|
||||
OR: [
|
||||
...(profile.phone ? [{ phone: profile.phone }] : []),
|
||||
{ email: profile.email },
|
||||
],
|
||||
},
|
||||
select: {
|
||||
id: true,
|
||||
status: true,
|
||||
@@ -229,7 +234,7 @@ async function provisionWecomOAuthEmployee({
|
||||
where: { id: existing.id },
|
||||
data: {
|
||||
name: profile.name,
|
||||
phone: profile.phone,
|
||||
...(profile.phone ? { phone: profile.phone } : {}),
|
||||
email: profile.email,
|
||||
department_id: departmentId,
|
||||
title: profile.title,
|
||||
@@ -241,7 +246,7 @@ async function provisionWecomOAuthEmployee({
|
||||
: await tx.employee.create({
|
||||
data: {
|
||||
name: profile.name,
|
||||
phone: profile.phone,
|
||||
phone: profile.phone || null,
|
||||
email: profile.email,
|
||||
department_id: departmentId,
|
||||
title: profile.title,
|
||||
@@ -294,7 +299,7 @@ async function provisionWecomOAuthEmployee({
|
||||
extra: {
|
||||
created: !existing,
|
||||
source: "wecom_oauth",
|
||||
fieldPresence: { phone: true, email: true },
|
||||
fieldPresence: { phone: Boolean(profile.phone), email: true },
|
||||
},
|
||||
}),
|
||||
});
|
||||
|
||||
@@ -219,6 +219,33 @@ describe("users import routes", () => {
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("accepts an import without a phone header and persists a missing phone as null", async () => {
|
||||
const { app, employeeCreate } = await buildApp({
|
||||
departments: [{ id: "dept-1", name: "行政部", archived_at: null }],
|
||||
});
|
||||
const multipartRequest = buildMultipartRequest({
|
||||
filename: "users.csv",
|
||||
contentType: "text/csv",
|
||||
content: [
|
||||
"employeeNo,name,email,departmentName,title,roles,hireDate",
|
||||
"E006,NoPhone,[email protected],行政部,员工,employee,2026-07-27",
|
||||
].join("\n"),
|
||||
});
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/users/import/apply",
|
||||
...multipartRequest,
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(employeeCreate).toHaveBeenCalledWith({
|
||||
data: expect.objectContaining({ phone: null }),
|
||||
select: expect.any(Object),
|
||||
});
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("rejects dry-run uploads with unsupported mime type", async () => {
|
||||
const { app } = await buildApp();
|
||||
const multipartRequest = buildMultipartRequest({
|
||||
|
||||
@@ -62,7 +62,27 @@ async function buildApp({
|
||||
roles: [{ role_code: "employee" }],
|
||||
},
|
||||
);
|
||||
const employeeUpdate = vi.fn();
|
||||
const employeeUpdate = vi.fn(async ({ data }) => ({
|
||||
...(currentEmployee ?? {
|
||||
id: USER_ID,
|
||||
employee_no: "HYD000321",
|
||||
name: "Alice",
|
||||
phone: "13800000000",
|
||||
email: "[email protected]",
|
||||
status: "active",
|
||||
department_id: DEPARTMENT_ID,
|
||||
title: "员工",
|
||||
direct_manager_id: null,
|
||||
hire_date: new Date("2026-05-01T00:00:00.000Z"),
|
||||
archived_at: null,
|
||||
created_at: new Date("2026-05-01T00:00:00.000Z"),
|
||||
updated_at: new Date("2026-05-01T00:00:00.000Z"),
|
||||
roles: [{ role_code: "employee" }],
|
||||
wecom_identity_binding: null,
|
||||
wecom_directory_sync_runs: [],
|
||||
}),
|
||||
...data,
|
||||
}));
|
||||
const employeeRoleCreateMany = vi.fn().mockResolvedValue({ count: 1 });
|
||||
const employeeRoleFindMany = vi.fn().mockResolvedValue(currentRoles.map((role_code) => ({ role_code })));
|
||||
const employeeRoleDeleteMany = vi.fn().mockResolvedValue({ count: currentRoles.length });
|
||||
@@ -139,10 +159,6 @@ async function buildApp({
|
||||
version: 1,
|
||||
}),
|
||||
},
|
||||
wecomDirectorySyncRun: {
|
||||
create: wecomSyncRunCreate,
|
||||
},
|
||||
wecomIdentityBinding: {},
|
||||
$transaction: vi.fn(async (callback) => callback(tx)),
|
||||
});
|
||||
app.decorate("requirePermission", (permissionCode) => async (request) => {
|
||||
@@ -243,6 +259,45 @@ describe("usersRoutes write flows", () => {
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("creates a user with a null phone when the field is omitted", async () => {
|
||||
const { app, employeeCreate } = await buildApp();
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/users",
|
||||
payload: {
|
||||
name: "暂缺手机号成员",
|
||||
email: "[email protected]",
|
||||
roles: ["employee"],
|
||||
syncWecomDirectory: false,
|
||||
},
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(employeeCreate).toHaveBeenCalledWith(expect.objectContaining({
|
||||
data: expect.objectContaining({ phone: null }),
|
||||
}));
|
||||
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("allows an administrator to explicitly clear an existing phone to null", async () => {
|
||||
const { app, employeeUpdate } = await buildApp();
|
||||
|
||||
const response = await app.inject({
|
||||
method: "PATCH",
|
||||
url: `/api/users/${USER_ID}`,
|
||||
payload: { phone: null },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(employeeUpdate).toHaveBeenCalledWith(expect.objectContaining({
|
||||
data: expect.objectContaining({ phone: null }),
|
||||
}));
|
||||
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("rejects employeeNo changes on patch", async () => {
|
||||
const { app, employeeFindUnique, employeeUpdate } = await buildApp();
|
||||
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
const ExcelJS = require("exceljs");
|
||||
|
||||
const {
|
||||
DirectoryExportError,
|
||||
buildSafeReport,
|
||||
parseArgs,
|
||||
parseDirectoryExport,
|
||||
} = require("../../../src/scripts/import-wecom-directory-export");
|
||||
|
||||
async function buildDirectoryWorkbook(rows) {
|
||||
const workbook = new ExcelJS.Workbook();
|
||||
const worksheet = workbook.addWorksheet("通讯录");
|
||||
worksheet.addRow(["填写须知：本行为企微导出说明"]);
|
||||
worksheet.addRow([]);
|
||||
worksheet.addRow([
|
||||
"姓名", "账号", "别名", "职务", "部门", "性别", "手机", "座机",
|
||||
"邮箱", "地址", "企业邮箱", "视频号", "激活状态", "禁用状态", "微信插件",
|
||||
]);
|
||||
for (const row of rows) {
|
||||
worksheet.addRow([
|
||||
row.name,
|
||||
row.account,
|
||||
"",
|
||||
row.title ?? "员工",
|
||||
row.department ?? "总部",
|
||||
"",
|
||||
row.phone ?? "",
|
||||
"",
|
||||
row.email ?? "",
|
||||
"",
|
||||
row.businessEmail,
|
||||
"",
|
||||
"已激活",
|
||||
"",
|
||||
"未关注",
|
||||
]);
|
||||
}
|
||||
return Buffer.from(await workbook.xlsx.writeBuffer());
|
||||
}
|
||||
|
||||
describe("WeCom directory export importer", () => {
|
||||
it("requires stdin, the apply environment gate, and an actor for apply", () => {
|
||||
const previous = process.env.WECOM_DIRECTORY_EXPORT_APPLY;
|
||||
delete process.env.WECOM_DIRECTORY_EXPORT_APPLY;
|
||||
try {
|
||||
expect(() => parseArgs(["--input=directory.xlsx"])).toThrowError(
|
||||
expect.objectContaining({ code: "stdin_required" }),
|
||||
);
|
||||
expect(() => parseArgs(["--apply", "--actor-employee-no=HYD000001"])).toThrowError(
|
||||
expect.objectContaining({ code: "apply_confirmation_required" }),
|
||||
);
|
||||
process.env.WECOM_DIRECTORY_EXPORT_APPLY = "1";
|
||||
expect(() => parseArgs(["--apply"])).toThrowError(
|
||||
expect.objectContaining({ code: "actor_required" }),
|
||||
);
|
||||
expect(parseArgs(["--apply", "--actor-employee-no=HYD000001"])).toEqual(
|
||||
expect.objectContaining({ input: "-", mode: "apply", actorEmployeeNo: "HYD000001" }),
|
||||
);
|
||||
} finally {
|
||||
if (previous == null) delete process.env.WECOM_DIRECTORY_EXPORT_APPLY;
|
||||
else process.env.WECOM_DIRECTORY_EXPORT_APPLY = previous;
|
||||
}
|
||||
});
|
||||
|
||||
it("finds the official Chinese header after instruction rows and accepts one missing phone", async () => {
|
||||
const rows = Array.from({ length: 6 }, (_, index) => ({
|
||||
name: `合成员${index + 1}`,
|
||||
account: `member-${index + 1}`,
|
||||
phone: index === 5 ? "" : `1990000000${index + 1}`,
|
||||
businessEmail: `member-${index + 1}@example.test`,
|
||||
}));
|
||||
const parsed = await parseDirectoryExport(await buildDirectoryWorkbook(rows));
|
||||
|
||||
expect(parsed.sourceSummary).toEqual({
|
||||
memberCount: 6,
|
||||
activeCount: 6,
|
||||
phonePresentCount: 5,
|
||||
phoneMissingCount: 1,
|
||||
emailPresentCount: 6,
|
||||
incompleteCount: 0,
|
||||
});
|
||||
expect(parsed.profiles[5].phone).toBeNull();
|
||||
expect(parsed.profiles[5].missingRequiredFields).toEqual([]);
|
||||
|
||||
const report = buildSafeReport(
|
||||
{ mode: "dry_run" },
|
||||
parsed,
|
||||
{ summary: { created: 5, updated: 1, planned: 6 } },
|
||||
);
|
||||
const serialized = JSON.stringify(report);
|
||||
expect(report.status).toBe("PASS_DIRECTORY_EXPORT_DRY_RUN");
|
||||
for (const row of rows) {
|
||||
expect(serialized).not.toContain(row.name);
|
||||
expect(serialized).not.toContain(row.account);
|
||||
expect(serialized).not.toContain(row.businessEmail);
|
||||
if (row.phone) expect(serialized).not.toContain(row.phone);
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
["account", { account: "member-1", phone: "19900000002", businessEmail: "[email protected]" }],
|
||||
["phone", { account: "member-2", phone: "19900000001", businessEmail: "[email protected]" }],
|
||||
["email", { account: "member-2", phone: "19900000002", businessEmail: "[email protected]" }],
|
||||
])("rejects duplicate %s values without returning the duplicated value", async (_field, duplicateRow) => {
|
||||
const buffer = await buildDirectoryWorkbook([
|
||||
{
|
||||
name: "合成员一",
|
||||
account: "member-1",
|
||||
phone: "19900000001",
|
||||
businessEmail: "[email protected]",
|
||||
},
|
||||
{ name: "合成员二", ...duplicateRow },
|
||||
]);
|
||||
|
||||
let caught;
|
||||
try {
|
||||
await parseDirectoryExport(buffer);
|
||||
} catch (error) {
|
||||
caught = error;
|
||||
}
|
||||
expect(caught).toBeInstanceOf(DirectoryExportError);
|
||||
expect(caught.code).toBe("duplicate_values");
|
||||
const serialized = JSON.stringify(caught);
|
||||
expect(serialized).not.toContain("[email protected]");
|
||||
expect(serialized).not.toContain("19900000001");
|
||||
});
|
||||
|
||||
it("requires a valid enterprise email even when a personal email is present", async () => {
|
||||
const parsed = await parseDirectoryExport(await buildDirectoryWorkbook([{
|
||||
name: "合成员一",
|
||||
account: "member-1",
|
||||
phone: "19900000001",
|
||||
email: "[email protected]",
|
||||
businessEmail: "",
|
||||
}]));
|
||||
|
||||
expect(parsed.sourceSummary.incompleteCount).toBe(1);
|
||||
expect(parsed.profiles[0].missingRequiredFields).toContain("email");
|
||||
});
|
||||
|
||||
it("blocks a supplied invalid phone instead of treating it as an absent phone", async () => {
|
||||
const parsed = await parseDirectoryExport(await buildDirectoryWorkbook([{
|
||||
name: "合成员一",
|
||||
account: "member-1",
|
||||
phone: "not-a-phone",
|
||||
businessEmail: "[email protected]",
|
||||
}]));
|
||||
|
||||
expect(parsed.sourceSummary.phoneMissingCount).toBe(1);
|
||||
expect(parsed.sourceSummary.incompleteCount).toBe(1);
|
||||
expect(parsed.profiles[0].missingRequiredFields).toContain("phone_invalid");
|
||||
});
|
||||
|
||||
it("uses a distinct success status for an apply report", async () => {
|
||||
const parsed = await parseDirectoryExport(await buildDirectoryWorkbook([{
|
||||
name: "合成员一",
|
||||
account: "member-1",
|
||||
phone: "",
|
||||
businessEmail: "[email protected]",
|
||||
}]));
|
||||
|
||||
const report = buildSafeReport(
|
||||
{ mode: "apply" },
|
||||
parsed,
|
||||
{ summary: { created: 1, success: 1 } },
|
||||
);
|
||||
expect(report.status).toBe("PASS_DIRECTORY_EXPORT_APPLY");
|
||||
});
|
||||
});
|
||||
@@ -5,6 +5,7 @@ const {
|
||||
evaluateAlertDecision,
|
||||
executeOpsAlert,
|
||||
parseArgs,
|
||||
readSummary,
|
||||
shouldSendForSummary,
|
||||
} = require("../../../src/scripts/send-wecom-ops-alert");
|
||||
|
||||
@@ -75,6 +76,72 @@ describe("send-wecom-ops-alert", () => {
|
||||
expect(message.body).not.toContain("手机号");
|
||||
});
|
||||
|
||||
it("reads stdin summary without preserving an injected absolute report path", () => {
|
||||
const summary = readSummary("-", JSON.stringify({
|
||||
issueId: "AUTO_BACKUP",
|
||||
status: "FAIL_AUTOMATIC_BACKUP",
|
||||
findingCount: 1,
|
||||
reportDir: `/private/${SENSITIVE_TOKEN}/state`,
|
||||
}));
|
||||
|
||||
expect(summary.issueId).toBe("AUTO_BACKUP");
|
||||
expect(summary.reportDir).toBe("stdin-summary");
|
||||
expect(JSON.stringify(summary)).not.toContain(SENSITIVE_TOKEN);
|
||||
expect(JSON.stringify(summary)).not.toContain("/private/");
|
||||
});
|
||||
|
||||
it("keeps AUTO_BACKUP issue id and uses backup-specific failure guidance", async () => {
|
||||
const summary = baseSummary({
|
||||
issueId: "AUTO_BACKUP",
|
||||
status: "FAIL_AUTOMATIC_BACKUP",
|
||||
reportDir: "host-backup-state",
|
||||
});
|
||||
const message = buildAlertMessage(summary);
|
||||
const report = await executeOpsAlert(baseOptions(), {
|
||||
summary,
|
||||
recipientRows: recipientRows(),
|
||||
sendMessageImpl: vi.fn(),
|
||||
env: {},
|
||||
config: {},
|
||||
});
|
||||
|
||||
expect(message.title).toContain("自动备份");
|
||||
expect(message.body).toContain("异机传输链路");
|
||||
expect(report.issueId).toBe("AUTO_BACKUP");
|
||||
expect(report.summary.issueId).toBe("AUTO_BACKUP");
|
||||
});
|
||||
|
||||
it("uses manual ARM shadow-restore guidance for quarterly reminder", () => {
|
||||
const summary = baseSummary({
|
||||
issueId: "AUTO_BACKUP",
|
||||
findings: [{ detail: "quarterly_shadow_restore_due_2026-Q3" }],
|
||||
reportDir: "host-backup-state",
|
||||
});
|
||||
const message = buildAlertMessage(summary);
|
||||
const firstDecision = evaluateAlertDecision(summary, baseOptions(), null);
|
||||
const repeatedDecision = evaluateAlertDecision(summary, baseOptions(), buildAlertState(summary));
|
||||
|
||||
expect(message.body).toContain("ARM 隔离目标执行人工影子恢复");
|
||||
expect(firstDecision.shouldSend).toBe(true);
|
||||
expect(firstDecision.reason).toBe("quarterly_restore_reminder");
|
||||
expect(repeatedDecision.shouldSend).toBe(false);
|
||||
});
|
||||
|
||||
it("sends the first automatic-backup failure and suppresses an identical repeat", () => {
|
||||
const summary = baseSummary({
|
||||
issueId: "AUTO_BACKUP",
|
||||
status: "FAIL_AUTOMATIC_BACKUP",
|
||||
findings: [{ source: "automatic_backup", status: "FAIL_AUTOMATIC_BACKUP", detail: "phase_offhost_transfer_failed" }],
|
||||
});
|
||||
const firstDecision = evaluateAlertDecision(summary, baseOptions(), null);
|
||||
const repeatedDecision = evaluateAlertDecision(summary, baseOptions(), buildAlertState(summary));
|
||||
|
||||
expect(firstDecision.shouldSend).toBe(true);
|
||||
expect(firstDecision.reason).toBe("blocked_or_failed");
|
||||
expect(repeatedDecision.shouldSend).toBe(false);
|
||||
expect(repeatedDecision.reason).toBe("repeated_automatic_backup_failure");
|
||||
});
|
||||
|
||||
it("keeps PASS without findings as no-send unless forced", () => {
|
||||
const summary = baseSummary({ status: "PASS_ARCHIVE_OPS_READY", findingCount: 0 });
|
||||
expect(shouldSendForSummary(summary, baseOptions())).toBe(false);
|
||||
|
||||
@@ -67,6 +67,38 @@ describe("wecom HR master sync helpers", () => {
|
||||
expect(profile.missingRequiredFields).toContain("email");
|
||||
});
|
||||
|
||||
it("accepts a profile with no phone and does not plan clearing an existing phone", () => {
|
||||
const profile = normalizeWecomHrProfile({
|
||||
contact: {
|
||||
userid: "missing-phone",
|
||||
name: "缺手机号成员",
|
||||
biz_mail: "[email protected]",
|
||||
department: [1],
|
||||
enable: 1,
|
||||
},
|
||||
staff: {},
|
||||
departmentMap: new Map([["1", { externalId: "1", name: "行政部" }]]),
|
||||
});
|
||||
|
||||
expect(profile.phone).toBe("");
|
||||
expect(profile.missingRequiredFields).toEqual([]);
|
||||
|
||||
const changes = __testables.buildEmployeeChanges(
|
||||
{
|
||||
id: "employee-1",
|
||||
name: profile.name,
|
||||
phone: "19900000009",
|
||||
email: profile.email,
|
||||
department_id: "department-1",
|
||||
title: "员工",
|
||||
wecom_identity_binding: { external_user_id: profile.externalUserId },
|
||||
},
|
||||
profile,
|
||||
{ id: "department-1", name: "行政部" },
|
||||
);
|
||||
expect(changes).not.toHaveProperty("phone");
|
||||
});
|
||||
|
||||
it("marks inactive WeCom members as confirmation-required instead of active", () => {
|
||||
const profile = normalizeWecomHrProfile({
|
||||
contact: {
|
||||
|
||||
@@ -101,6 +101,20 @@ describe("wecom OAuth onboarding service", () => {
|
||||
});
|
||||
expect(profile.email).toBe("[email protected]");
|
||||
|
||||
const profileWithoutPhone = await fetchWecomOAuthOnboardingProfile({
|
||||
accessToken: "agent-token",
|
||||
externalUserId: "member-fields",
|
||||
userTicket: "ticket",
|
||||
fetchImpl: buildFetch({
|
||||
errcode: 0,
|
||||
userid: "member-fields",
|
||||
mobile: "",
|
||||
biz_mail: "[email protected]",
|
||||
}),
|
||||
});
|
||||
expect(profileWithoutPhone.phone).toBe("");
|
||||
expect(profileWithoutPhone.email).toBe("[email protected]");
|
||||
|
||||
await expect(fetchWecomOAuthOnboardingProfile({
|
||||
accessToken: "agent-token",
|
||||
externalUserId: "member-fields",
|
||||
@@ -238,6 +252,52 @@ describe("wecom OAuth onboarding service", () => {
|
||||
expect(persisted).not.toContain("[email protected]");
|
||||
});
|
||||
|
||||
it("creates a member with a null phone when the authorized profile has no phone", async () => {
|
||||
const tx = {
|
||||
wecomIdentityBinding: {
|
||||
findUnique: vi.fn().mockResolvedValue(null),
|
||||
upsert: vi.fn().mockResolvedValue({ id: "binding-1" }),
|
||||
},
|
||||
employee: {
|
||||
findMany: vi.fn().mockResolvedValue([]),
|
||||
create: vi.fn().mockResolvedValue({ id: "employee-1", token_version: 0 }),
|
||||
},
|
||||
integrationBinding: {
|
||||
findFirst: vi.fn().mockResolvedValue({ local_id: "department-1" }),
|
||||
},
|
||||
department: {
|
||||
findUnique: vi.fn().mockResolvedValue({ id: "department-1", archived_at: null }),
|
||||
},
|
||||
employeeRole: { create: vi.fn().mockResolvedValue({}) },
|
||||
auditLog: { create: vi.fn().mockResolvedValue({}) },
|
||||
};
|
||||
const prisma = { $transaction: vi.fn(async (callback) => callback(tx)) };
|
||||
|
||||
await provisionWecomOAuthEmployee({
|
||||
prisma,
|
||||
request: buildRequest(),
|
||||
profile: {
|
||||
externalUserId: "member-no-phone",
|
||||
name: "缺手机号成员",
|
||||
phone: "",
|
||||
email: "[email protected]",
|
||||
departmentExternalId: "1",
|
||||
departmentName: "总部",
|
||||
title: "员工",
|
||||
},
|
||||
});
|
||||
|
||||
expect(tx.employee.create).toHaveBeenCalledWith({
|
||||
data: expect.objectContaining({ phone: null }),
|
||||
select: expect.any(Object),
|
||||
});
|
||||
expect(tx.auditLog.create).toHaveBeenCalledWith({
|
||||
data: expect.objectContaining({
|
||||
extra: expect.objectContaining({ fieldPresence: { phone: false, email: true } }),
|
||||
}),
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects contact data that resolves to multiple OA employees", async () => {
|
||||
const tx = {
|
||||
wecomIdentityBinding: { findUnique: vi.fn().mockResolvedValue(null) },
|
||||
|
||||
在新工单中引用
屏蔽一个用户