优化(wecom-public-health-alert-diagnostics-and-debounce): 增强公网健康告警诊断与防抖

.trellis/spec/hyb-oa/deployment-trialrun.md

@@ -1443,18 +1443,42 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
 - Units: `hyb-oa-public-app-health.service` and
   `hyb-oa-public-app-health.timer`; the timer uses `OnCalendar=*:0/5`.
 - Local contract: `bash scripts/qa/run-app-health-monitor-contract.sh`.
+- Application statuses: `FAIL_APP_HEALTH`, `PASS_APP_HEALTH_READY` and
+  `FAIL_APP_HEALTH_FLAPPING`.
+- Probe-state schema v2 phases: `healthy`, `failure_pending`, `alerted`,
+  `recovery_pending`, `recovery_watch` and `flapping`.
 
 ### 3. Contracts
 
 - Run the probe from the existing ARM host against an exact HTTPS `/ready` URL.
   Success requires trusted TLS, HTTP 2xx, bounded JSON and `data.ok === true`.
-- Count any stable failure category toward the same consecutive-failure
-  threshold. The first failure is pending; the second evaluates a failure
-  alert. Once alerting, a changed category may send a new alert, identical
-  failures are deduplicated, and one later success sends one recovery notice.
-- Persist only schema version, count, stable category and the alert state in a
-  root-owned `0700` directory with atomic `0600` files. Never persist or send
-  the URL, resolved address, response payload, credential or raw exception.
+- Classify DNS resolution and TCP connection failures separately before the
+  shared fallback catches `OSError`; keep TLS, timeout, HTTP, response-size and
+  response-contract categories stable. Count any stable failure category toward
+  the same consecutive-failure threshold. The first failure is pending and the
+  second evaluates a failure alert; identical failures remain deduplicated.
+- After an alert, require two consecutive healthy samples before emitting one
+  recovery. A single healthy sample followed by another failure stays in the
+  existing incident and sends no recovery noise.
+- Keep the recovered category under a 60-minute observation window. The same
+  category reaching the two-failure threshold in that window emits one
+  `FAIL_APP_HEALTH_FLAPPING` reminder. Later switches stay local until health
+  has remained continuous for 30 minutes, then the probe exits flapping
+  silently. A different category in the recovery window remains a new normal
+  incident and still uses the two-failure threshold.
+- Persist only schema version, phase, bounded counts, stable categories and
+  necessary Unix timestamps in a root-owned `0700` directory with atomic
+  `0600` files. Schema v1 migrates conservatively without backfilling a
+  notification. Never persist or send the URL, resolved address, response
+  payload, credential or raw exception.
+- The probe returns a notification candidate on repeated samples while the
+  corresponding state is active; the shared helper owns final deduplication.
+  This lets a failed send retry on a later sample without repeating a
+  successfully sent message. Healthy flapping samples may retry the stable
+  flapping reminder but never produce a recovery candidate.
+- Application messages use the explicit titles `OA 公网健康告警`,
+  `OA 公网健康恢复` and `OA 公网健康链路反复抖动`. Shared backup/dead-man
+  statuses retain their generic operations titles.
 - Application-health probe state and alert state are separate from backup and
   dead-man state. The installer must deploy the helper revision that passed the
   backup contract; an older installed helper may reject the new status tokens.
@@ -1474,11 +1498,19 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
 
 - Non-HTTPS, userinfo, query, fragment or non-`/ready` URL -> reject before a
   request with `ready_url_invalid`.
-- DNS/connect, TLS, timeout, HTTP, oversized response or invalid JSON/contract
-  -> stable redacted category and non-zero oneshot exit.
+- DNS, TCP connection, TLS, timeout, HTTP, oversized response or invalid
+  JSON/contract -> distinct stable redacted category and non-zero oneshot exit.
 - Invalid owner/mode/link/state or alert-helper failure -> fail closed with a
   fixed local error; do not include the offending value.
 - A first isolated failure followed by health -> no failure or recovery message.
+- One healthy sample after an alerted failure -> `recovery_pending`, no recovery
+  message; the second consecutive sample -> one recovery.
+- Same-category recurrence before 60 minutes -> one flapping reminder; at
+  exactly 60 minutes -> a new normal incident, not flapping.
+- Flapping health shorter than 30 minutes -> remain flapping; at 30 continuous
+  minutes -> return to `healthy` without a message.
+- Invalid/unknown schema, out-of-range counter/timestamp or inconsistent phase
+  fields -> `local_state_invalid`, with no raw state in output.
 - Any overridden managed target or service name -> installer stops before
   `systemctl`, file replacement or removal.
 - SSH succeeds but `sudo -n` requests a password -> stop before upload or
@@ -1491,8 +1523,9 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
 
 ### 5. Good/Base/Bad Cases
 
-- Good: ARM observes two consecutive unavailable checks, sends one redacted
-  category alert, suppresses repeats and sends one recovery after health returns.
+- Good: ARM observes two consecutive unavailable checks, sends one explicit
+  OA-public-health alert, suppresses repeats, confirms recovery with two healthy
+  samples, and emits only one flapping reminder for a same-category recurrence.
 - Base: repository contracts pass while units remain disabled; report the code
   as ready for an approved rollout, not as an active monitor.
 - Bad: run the timer only on the OA host, send raw `/ready` errors, install and
@@ -1501,9 +1534,15 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
 ### 6. Tests Required
 
 - The local HTTPS stub must assert healthy baseline, pending first failure,
-  second-failure alert, identical deduplication, changed-category alert, one
-  recovery, transient suppression, TLS/connect/timeout/HTTP/size/JSON failures,
-  corrupt private state, URL rejection and absence of sensitive output.
+  second-failure alert, identical deduplication, changed-category alert,
+  two-sample recovery, transient suppression, same-category flapping,
+  different-category recurrence, silent 30-minute exit, exact 30/60-minute
+  boundaries, schema-v1 migration, separate DNS/TCP classification,
+  TLS/timeout/HTTP/size/JSON failures, corrupt private state, URL rejection and
+  absence of sensitive output.
+- Captured application messages must assert the three explicit titles and
+  stable status/category only. The automatic-backup contract must prove shared
+  helper titles, recovery and deduplication remain unchanged.
 - Run Python compile and shell syntax checks, validate the five-minute calendar,
   and assert disabled-by-default plus exact managed targets and retained state.
 - For a remote rollout, record boolean evidence for pre-write `sudo -n`, exact
@@ -1520,7 +1559,9 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
 #### Wrong
 
 ```bash
-# Local contract passed, so install and enable remotely and call it active.
+# One healthy sample arrived, so announce recovery immediately and keep
+# alternating alert/recovery messages during a flapping link.
+bash scripts/qa/run-app-health-monitor-contract.sh
 bash deploy/trialrun/install-offhost-app-health-monitor.sh enable
 ```
 
@@ -1529,6 +1570,8 @@ bash deploy/trialrun/install-offhost-app-health-monitor.sh enable
 ```bash
 bash scripts/qa/run-app-health-monitor-contract.sh
 bash scripts/qa/run-automatic-backup-contract.sh
+# Runtime contract: fail x2 -> alert; healthy x2 -> recovery; same fail x2
+# within 60m -> one flapping reminder; continuous health 30m -> silent exit.
 # After separate ARM-write approval: install disabled, inspect, then seek
 # enable/real-message/drill approval before claiming the monitor is active.
 <explicit-controlled-paths> \