验证(08-23-isolated-backup-restore-drill): 固化隔离恢复runner合同
这个提交包含在：
@@ -1210,6 +1210,256 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
|
||||
"$(printf '%s\n' "$expected" | LC_ALL=C sort)"
|
||||
```
|
||||
|
||||
## Scenario: Off-Host Archive And Private-Copy Identity Validation
|
||||
|
||||
### 1. Scope / Trigger
|
||||
|
||||
- Trigger: an off-host recovery drill copies a formally archived backup batch
|
||||
into a private temporary directory before database or attachment restore.
|
||||
- Purpose: preserve the archive placement evidence at the source boundary while
|
||||
validating a renamed private copy by its source-bound content identity.
|
||||
|
||||
### 2. Signatures
|
||||
|
||||
- Source validation:
|
||||
`validate_source_batch(<archive-run-directory>) -> source_metadata`.
|
||||
- Private-copy validation:
|
||||
`validate_copied_batch(<private-copy-directory>, <source_metadata>)`.
|
||||
- `source_metadata` contains only redacted comparison values: verified run id,
|
||||
manifest/build/time identity, exact three-file set, byte sizes and SHA-256
|
||||
values. It must not contain credentials, business payloads or full paths.
|
||||
|
||||
### 3. Contracts
|
||||
|
||||
- A formal archive run directory must have a basename that exactly equals the
|
||||
validated manifest run id. This detects a valid manifest placed under the
|
||||
wrong archive run and is mandatory before any copy or restore write.
|
||||
- A private copy may use an operator-owned neutral basename such as `batch`;
|
||||
that basename is not archive provenance and must not be compared with the
|
||||
manifest run id.
|
||||
- The copied manifest run id must exactly equal the run id already verified at
|
||||
the source. Internal self-consistency inside the copy is insufficient.
|
||||
- The copy must repeat manifest schema/status, run-id format, exact
|
||||
`manifest.json`/`database.dump`/`storage.tar.gz` regular-file set, build and
|
||||
age gates, byte-size checks and SHA-256 checks.
|
||||
- After repeated validation, every redacted copied metadata field must equal
|
||||
`source_metadata`. Copy validation must never disable a source-side gate or
|
||||
accept a newly invented identity.
|
||||
- Formal archives remain read-only. A validation failure stops before creating
|
||||
PostgreSQL, extracting attachments or modifying the archive, and cleanup must
|
||||
prove zero private runtime and privilege-entry residue.
|
||||
|
||||
### 4. Validation & Error Matrix
|
||||
|
||||
- Source basename differs from manifest run id -> `batch_identity`; `NO_GO`
|
||||
before copy.
|
||||
- Copied manifest run id differs from the verified source run id, even when the
|
||||
copy is internally self-consistent -> `copied_batch_invalid/batch_identity`;
|
||||
`NO_GO` before restore runtime creation.
|
||||
- Copied file set, bytes, SHA-256, build, time or other redacted metadata differs
|
||||
from the source -> `copied_batch_invalid`; remove only the owned private
|
||||
runtime and retain the archive unchanged.
|
||||
- Private basename differs from the run id but all source-bound content checks
|
||||
pass -> valid; continue to the separately approved restore step.
|
||||
- Any cleanup residue remains -> attempt stays non-clean and cannot be reported
|
||||
as recovery `GO`.
|
||||
|
||||
### 5. Good/Base/Bad Cases
|
||||
|
||||
- Good: source directory basename matches its manifest run id; the same batch
|
||||
is copied to a private directory named `batch`, revalidated, and its redacted
|
||||
metadata exactly equals the source record.
|
||||
- Base: source and copy basenames happen to match; both still use their proper
|
||||
validation modes and the copy remains bound to `source_metadata`.
|
||||
- Bad: reuse source validation on the private directory and reject it only
|
||||
because the operator chose a neutral basename.
|
||||
- Bad: skip the copy basename check and then trust any internally consistent
|
||||
replacement manifest without comparing it with the verified source run id.
|
||||
|
||||
### 6. Tests Required
|
||||
|
||||
- Positive: a formal source whose basename matches its manifest run id passes.
|
||||
- Positive: an unchanged copy under a different private basename passes and
|
||||
produces metadata equal to the source record.
|
||||
- Negative: source basename/run-id mismatch fails before copy.
|
||||
- Negative: copied content or checksum mutation fails before restore.
|
||||
- Negative: a copied manifest with an internally consistent but source-different
|
||||
run id fails, proving the source binding cannot be bypassed.
|
||||
- Regression: run the existing synthetic backup/restore contract and assert no
|
||||
PostgreSQL object, extracted attachment, private runtime or privilege entry is
|
||||
left after every pre-runtime failure.
|
||||
|
||||
### 7. Wrong vs Correct
|
||||
|
||||
#### Wrong
|
||||
|
||||
```text
|
||||
validate_batch(source, require_basename_match=true)
|
||||
copy source -> private/batch
|
||||
validate_batch(private/batch, require_basename_match=true)
|
||||
```
|
||||
|
||||
#### Correct
|
||||
|
||||
```text
|
||||
source_metadata = validate_source_batch(source_run_directory)
|
||||
copy source_run_directory -> private/batch
|
||||
validate_copied_batch(private/batch, source_metadata)
|
||||
```
|
||||
|
||||
## Scenario: Shadow-Restore Failure Diagnostics Before Teardown
|
||||
|
||||
### 1. Scope / Trigger
|
||||
|
||||
- Trigger: a one-shot off-host shadow restore keeps raw tool output only in a
|
||||
private runtime that must be deleted on every exit path.
|
||||
- Purpose: preserve enough allowlisted stage evidence to locate a restore
|
||||
failure without retaining or publishing raw stderr, paths, credentials, SQL
|
||||
or restored object names.
|
||||
|
||||
### 2. Signatures
|
||||
|
||||
- Required stages: `dump_list`, `target_confirmation`, `restore_apply`, each
|
||||
with the enum `PASS|FAIL|NOT_RUN`.
|
||||
- Required execution fields: `restore_execution_path=DOCKER_STDIN|HOST_FILE|NOT_RUN`,
|
||||
`dump_list_exit_code`, `restore_script_exit_code`,
|
||||
`pg_restore_exit_code`, `pg_restore_error_count` and
|
||||
`restore_error_category`.
|
||||
- Exit-code fields are non-negative integers or `NOT_RUN`;
|
||||
`pg_restore_error_count` additionally permits `UNKNOWN` when no safe numeric
|
||||
summary is available.
|
||||
- Tracked supply-chain entries:
|
||||
`scripts/qa/offhost-shadow-restore-runner.template.sh`,
|
||||
`scripts/qa/validate-offhost-shadow-restore-attachments.py` and
|
||||
`bash scripts/qa/run-offhost-shadow-restore-runner-contract.sh`.
|
||||
- Candidate render:
|
||||
`bash scripts/qa/run-offhost-shadow-restore-runner-contract.sh render <private-config.json> <new-output>`;
|
||||
it prints only PASS and the candidate SHA-256, never config values or paths.
|
||||
|
||||
### 3. Contracts
|
||||
|
||||
- Emit the fixed diagnostic fields on the failure path before runtime teardown.
|
||||
Teardown remains in a trap/finally-equivalent path and must still run after
|
||||
diagnostic emission; callers retain stdout, not the private raw log.
|
||||
- An ARM restore runner fixes `restore_execution_path=DOCKER_STDIN`. A private
|
||||
Docker proxy may observe the real `pg_restore --list` and apply calls and
|
||||
persist only their stage name and numeric exit code; it must not record argv,
|
||||
environment values or stderr.
|
||||
- Run `dump_list` and target confirmation independently before the existing
|
||||
restore script. A failed stage leaves every later stage `NOT_RUN`.
|
||||
- Raw restore output remains mode `0600`, never leaves the off-host node and is
|
||||
deleted with the owned runtime on success and failure.
|
||||
- Any non-zero pg_restore apply exit is `FAIL/NO_GO`, even when a numeric
|
||||
"errors ignored" count can be extracted. There is no partial-success
|
||||
allowlist. Relaxing that rule requires a separately approved design proving
|
||||
the exact restored object set.
|
||||
- A diagnostic failure is an exercise-runner finding until evidence identifies
|
||||
a dump or restore-content defect. It must not be summarized as proof that the
|
||||
latest backup is unrecoverable.
|
||||
- The tracked runner is a template, not a site-specific script. Remote address,
|
||||
account, paths, container/database names, timer units, build identity and
|
||||
migration fingerprint arrive only through a private JSON with an exact key
|
||||
set. Rendering validates every value's type, charset and safety shape before
|
||||
exclusively creating a candidate; runtime environment variables cannot
|
||||
override those rendered constants.
|
||||
- The exact rendered candidate must pass the tracked contract and preserve a
|
||||
deterministic SHA-256 before upload. A workspace-only runner, hand-edited
|
||||
candidate or contract failure is a hard upload blocker.
|
||||
- Private render JSON and site-specific candidates must live outside the
|
||||
repository or under the dedicated ignored `/.restore-drill-private/`
|
||||
directory. The renderer checks both input and output with Git ignore rules
|
||||
and fails before candidate creation when an in-repository path is not
|
||||
ignored; operator memory is not a control.
|
||||
- The mode-`0600` raw log is truncated exactly once at initialization. Container
|
||||
start, dump list, target confirmation, restore apply and storage restore each
|
||||
append a fixed stage marker and their private output, so a later failure does
|
||||
not erase earlier-stage diagnostics before final teardown.
|
||||
|
||||
### 4. Validation & Error Matrix
|
||||
|
||||
- Dump list non-zero -> `dump_list=FAIL`, later stages `NOT_RUN`, category
|
||||
`DUMP_LIST_NONZERO` / `dump_list_failed`.
|
||||
- Target query non-zero or exact shadow name mismatch ->
|
||||
`target_confirmation=FAIL`, apply `NOT_RUN`, category
|
||||
`TARGET_CONFIRMATION_FAILED`.
|
||||
- Independent list and target pass, but the restore script never invokes apply
|
||||
-> category `APPLY_EXIT_CODE_MISSING`; fail closed.
|
||||
- Apply exit non-zero -> `restore_apply=FAIL`, preserve the exact numeric exit
|
||||
and safe numeric/`UNKNOWN` error count, category `RESTORE_APPLY_NONZERO`.
|
||||
- Apply and script exit zero -> `restore_apply=PASS`; later migration, table and
|
||||
attachment checks still decide the final recovery result.
|
||||
- Diagnostics emitted only after cleanup, or cleanup residue non-zero -> the
|
||||
attempt is non-clean and cannot be reported as `GO`.
|
||||
- Missing/extra render key, newline/shell syntax, unsafe path/name, malformed
|
||||
digest or pre-existing output -> render fails without publishing a partial
|
||||
candidate.
|
||||
- In-repository private config or candidate path not covered by Git ignore ->
|
||||
render fails before the candidate exists. The dedicated ignored directory
|
||||
remains valid for a locally protected render.
|
||||
- Any raw-log truncating redirect after initialization, or a missing fixed stage
|
||||
marker -> tracked contract failure; do not publish a candidate.
|
||||
- Candidate differs from the deterministic render or has not passed the tracked
|
||||
contract -> do not upload or install it.
|
||||
|
||||
### 5. Good/Base/Bad Cases
|
||||
|
||||
- Good: dump list and target pass, apply exits `23`; fixed fields show
|
||||
`DOCKER_STDIN`, `23` and `RESTORE_APPLY_NONZERO` before teardown, then cleanup
|
||||
reports zero residue without exposing the raw error.
|
||||
- Base: all three stages pass with exit code `0`; the runner continues to
|
||||
migration and attachment verification.
|
||||
- Bad: emit only `database_restore_failed` after deleting the sole private log.
|
||||
- Bad: parse and publish arbitrary pg_restore lines, or treat a non-zero exit as
|
||||
success because the restored database appears partly populated.
|
||||
- Bad: commit one host's concrete runner or reconstruct a supposedly equivalent
|
||||
runner from chat/workspace immediately before an exercise.
|
||||
|
||||
### 6. Tests Required
|
||||
|
||||
- Directly drive the repository `scripts/restore-db.sh` with a real custom dump
|
||||
through Docker stdin and host-file execution paths; do not replace the script
|
||||
with an equivalent test implementation.
|
||||
- Inject list failure, target mismatch and list-pass/apply-failure. Assert exact
|
||||
stage enums, later-stage `NOT_RUN`, execution path and numeric exit code.
|
||||
- Inject a non-zero apply exit and assert the diagnostic is emitted before
|
||||
teardown, teardown passes, and the calling test can still read the fixed
|
||||
fields afterwards.
|
||||
- Scan captured diagnostics for fixture credentials, connection strings, SQL,
|
||||
temporary paths and free-form tool output.
|
||||
- Run the existing synthetic paired backup/restore and automatic off-host
|
||||
contract suites after changing the diagnostic runner.
|
||||
- Render the same private fixture twice and require byte equality; inject shell
|
||||
syntax and require failure before the output file exists.
|
||||
- Assert an unignored in-repository private config and an unignored candidate
|
||||
path both fail before output creation; assert the dedicated ignored directory
|
||||
accepts synthetic input and output without becoming trackable.
|
||||
- Statically require exactly one raw-log truncation at initialization and fixed
|
||||
append markers for container start, dump list, target confirmation, restore
|
||||
apply and storage restore.
|
||||
- Directly test source/copy identity, dynamic migration fingerprint and the
|
||||
tracked attachment validator. Run `check-sensitive-content.mjs` explicitly on
|
||||
all three tracked runner assets and require zero findings.
|
||||
|
||||
### 7. Wrong vs Correct
|
||||
|
||||
#### Wrong
|
||||
|
||||
```text
|
||||
restore-db > private.log || fail database_restore_failed
|
||||
finally: delete private runtime
|
||||
print result=NO_GO
|
||||
```
|
||||
|
||||
#### Correct
|
||||
|
||||
```text
|
||||
tracked template + exact private config -> deterministic candidate + SHA-256
|
||||
contract PASS -> list -> target confirmation -> real restore path
|
||||
capture only stage/path/exit/category enums
|
||||
emit fixed diagnostics -> finally teardown raw private runtime
|
||||
non-zero apply remains FAIL
|
||||
```
|
||||
|
||||
## Scenario: Daily Restricted Off-Host Backup
|
||||
|
||||
### 1. Scope / Trigger
|
||||
|
||||
在新工单中引用
屏蔽一个用户