实现应用健康监控（07-27-app-health-monitoring）
这个提交包含在：
@@ -57,6 +57,14 @@
|
||||
- [ ] **EXTERNAL_GATED**：已真正停止一次源端 timer 或故意破坏一次传输，ARM dead-man 仅产生一条脱敏企微告警；恢复 finalize 后收到一次恢复通知，同一失败重复运行已降噪
|
||||
- [ ] 若调整过 `BACKUP_RUN_AT`，已人工确认调整后的次日 UTC 分桶和 26 小时窗口，未把一次预期空窗误判为长期故障
|
||||
- [ ] 本季度已由第二名内部人员按 runbook 首部恢复卡在 ARM 隔离目标完成影子恢复走查；timer 未自动恢复生产
|
||||
- [ ] ARM 公网应用健康监控（未完成远端门禁前保持未勾选）：
|
||||
- [ ] 本地 `bash scripts/qa/run-app-health-monitor-contract.sh` 通过，且自动备份告警契约回归通过
|
||||
- [ ] **EXTERNAL_GATED**：已批准 ARM systemd 写入，并确认复用 root-owned `0600` webhook；安装的是已通过自动备份回归的共用 helper 修订，不复制或输出 webhook 正文
|
||||
- [ ] `install-offhost-app-health-monitor.sh install` 只安装 disabled units；应用健康 state/alert state 与备份 dead-man 完全独立
|
||||
- [ ] 手工 oneshot 健康执行通过，输出不含地址、路径、响应 payload、凭证或原始异常
|
||||
- [ ] **EXTERNAL_GATED**：另获批准后执行 `enable`，timer 为 enabled/active，oneshot service 平时 inactive，下一次运行不超过 5 分钟
|
||||
- [ ] **EXTERNAL_GATED**：持续失败演练中第一次仅 pending、第二次只发一条、重复失败降噪、恢复只通知一次；未用“机器人能发消息”替代故障链路
|
||||
- [ ] 已记录剩余风险：ARM 自身失效会使探测失声，首版没有第三方 dead-man
|
||||
- [ ] 停机复制方案已冻结：旧 PostgreSQL 完全停止后复制到空新目录并保留所有权/权限/链接/扩展属性；旧目录原位保留且观察期内不再写入
|
||||
|
||||
## B. 发布（Deploy）
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
READY_URL="${READY_URL:-https://oa.hyinda.com/ready}"
|
||||
PROBE_SOURCE="${PROBE_SOURCE:-}"
|
||||
PROBE_TARGET="${PROBE_TARGET:-/usr/local/sbin/hyb-oa-public-app-health}"
|
||||
ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}"
|
||||
ALERT_HELPER_SOURCE="${ALERT_HELPER_SOURCE:-}"
|
||||
ALERT_HELPER_TARGET="${ALERT_HELPER_TARGET:-/usr/local/sbin/hyb-oa-wecom-group-alert}"
|
||||
STATE_DIR="${STATE_DIR:-/var/lib/hyb-oa-app-health}"
|
||||
FAILURE_THRESHOLD="${FAILURE_THRESHOLD:-2}"
|
||||
TIMEOUT_SECONDS="${TIMEOUT_SECONDS:-10}"
|
||||
SERVICE_NAME="${SERVICE_NAME:-hyb-oa-public-app-health}"
|
||||
SYSTEMD_DIR="${SYSTEMD_DIR:-/etc/systemd/system}"
|
||||
MODE="${1:-install}"
|
||||
|
||||
fail() {
|
||||
echo "[app-health-installer][ERROR] $1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
need_root() {
|
||||
[[ "$(id -u)" == "0" ]] || fail "root privileges are required"
|
||||
}
|
||||
|
||||
validate_absolute_path() {
|
||||
local name="$1"
|
||||
local value="$2"
|
||||
if [[ -z "$value" || "$value" != /* || ! "$value" =~ ^/[A-Za-z0-9._/-]+$ ]]; then
|
||||
fail "$name must be an explicit safe absolute path"
|
||||
fi
|
||||
if [[ "$value" == "/" || "$value" == */ || "$value" == *"//"* \
|
||||
|| "/$value/" == *"/../"* || "/$value/" == *"/./"* ]]; then
|
||||
fail "$name contains an unsafe path segment"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_managed_targets() {
|
||||
[[ "$PROBE_TARGET" == "/usr/local/sbin/hyb-oa-public-app-health" ]] \
|
||||
|| fail "PROBE_TARGET must remain /usr/local/sbin/hyb-oa-public-app-health"
|
||||
[[ "$ALERT_HELPER_TARGET" == "/usr/local/sbin/hyb-oa-wecom-group-alert" ]] \
|
||||
|| fail "ALERT_HELPER_TARGET must remain /usr/local/sbin/hyb-oa-wecom-group-alert"
|
||||
[[ "$STATE_DIR" == "/var/lib/hyb-oa-app-health" ]] \
|
||||
|| fail "STATE_DIR must remain /var/lib/hyb-oa-app-health"
|
||||
[[ "$SYSTEMD_DIR" == "/etc/systemd/system" ]] \
|
||||
|| fail "SYSTEMD_DIR must remain /etc/systemd/system"
|
||||
[[ "$SERVICE_NAME" == "hyb-oa-public-app-health" ]] \
|
||||
|| fail "SERVICE_NAME must remain hyb-oa-public-app-health"
|
||||
}
|
||||
|
||||
validate_inputs() {
|
||||
local name value
|
||||
for name in PROBE_SOURCE PROBE_TARGET ALERT_WEBHOOK_FILE ALERT_HELPER_SOURCE ALERT_HELPER_TARGET STATE_DIR SYSTEMD_DIR; do
|
||||
value="${!name:-}"
|
||||
validate_absolute_path "$name" "$value"
|
||||
done
|
||||
validate_managed_targets
|
||||
[[ "$READY_URL" =~ ^https://[A-Za-z0-9.-]+(:([0-9]{1,5}))?/ready$ ]] \
|
||||
|| fail "READY_URL must be an HTTPS /ready endpoint without query or fragment"
|
||||
if [[ -n "${BASH_REMATCH[2]:-}" && ("${BASH_REMATCH[2]}" -lt 1 || "${BASH_REMATCH[2]}" -gt 65535) ]]; then
|
||||
fail "READY_URL port is invalid"
|
||||
fi
|
||||
[[ "$FAILURE_THRESHOLD" == "2" ]] || fail "FAILURE_THRESHOLD must remain 2"
|
||||
[[ "$TIMEOUT_SECONDS" =~ ^([1-9]|[12][0-9]|30)$ ]] || fail "TIMEOUT_SECONDS must be between 1 and 30"
|
||||
[[ -f "$PROBE_SOURCE" && ! -L "$PROBE_SOURCE" ]] || fail "probe source is missing"
|
||||
[[ -f "$ALERT_HELPER_SOURCE" && ! -L "$ALERT_HELPER_SOURCE" ]] || fail "alert helper source is missing"
|
||||
[[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \
|
||||
&& "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" \
|
||||
&& "$(stat -c %u "$ALERT_WEBHOOK_FILE")" == "0" ]] \
|
||||
|| fail "alert webhook file must be root-owned with mode 0600"
|
||||
[[ -d "$SYSTEMD_DIR" && ! -L "$SYSTEMD_DIR" ]] || fail "systemd directory is invalid"
|
||||
command -v python3 >/dev/null 2>&1 || fail "python3 is required"
|
||||
command -v systemctl >/dev/null 2>&1 || fail "systemctl is required"
|
||||
}
|
||||
|
||||
install_monitor() {
|
||||
need_root
|
||||
validate_inputs
|
||||
systemctl disable --now "${SERVICE_NAME}.timer" >/dev/null 2>&1 || true
|
||||
systemctl stop "${SERVICE_NAME}.service" >/dev/null 2>&1 || true
|
||||
install -o root -g root -m 0755 "$PROBE_SOURCE" "$PROBE_TARGET"
|
||||
install -o root -g root -m 0755 "$ALERT_HELPER_SOURCE" "$ALERT_HELPER_TARGET"
|
||||
mkdir -p "$STATE_DIR"
|
||||
[[ -d "$STATE_DIR" && ! -L "$STATE_DIR" ]] || fail "state directory is invalid"
|
||||
chown root:root "$STATE_DIR"
|
||||
chmod 0700 "$STATE_DIR"
|
||||
|
||||
cat >"$SYSTEMD_DIR/${SERVICE_NAME}.service" <<SERVICE
|
||||
[Unit]
|
||||
Description=hyb-oa independent public application readiness probe
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
UMask=0077
|
||||
ExecStart=/usr/bin/python3 ${PROBE_TARGET} --ready-url ${READY_URL} --state-dir ${STATE_DIR} --webhook-file ${ALERT_WEBHOOK_FILE} --alert-helper ${ALERT_HELPER_TARGET} --failure-threshold ${FAILURE_THRESHOLD} --timeout-seconds ${TIMEOUT_SECONDS}
|
||||
TimeoutStartSec=30
|
||||
SERVICE
|
||||
|
||||
cat >"$SYSTEMD_DIR/${SERVICE_NAME}.timer" <<TIMER
|
||||
[Unit]
|
||||
Description=Check hyb-oa public application readiness every five minutes
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*:0/5
|
||||
Persistent=true
|
||||
AccuracySec=10s
|
||||
Unit=${SERVICE_NAME}.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
TIMER
|
||||
chmod 0644 "$SYSTEMD_DIR/${SERVICE_NAME}.service" "$SYSTEMD_DIR/${SERVICE_NAME}.timer"
|
||||
systemctl daemon-reload
|
||||
echo "[app-health-installer][PASS] disabled application health monitor installed"
|
||||
}
|
||||
|
||||
enable_monitor() {
|
||||
install_monitor
|
||||
systemctl enable --now "${SERVICE_NAME}.timer"
|
||||
echo "[app-health-installer][PASS] application health timer enabled"
|
||||
}
|
||||
|
||||
uninstall_monitor() {
|
||||
need_root
|
||||
validate_absolute_path PROBE_TARGET "$PROBE_TARGET"
|
||||
validate_absolute_path SYSTEMD_DIR "$SYSTEMD_DIR"
|
||||
[[ "$PROBE_TARGET" == "/usr/local/sbin/hyb-oa-public-app-health" ]] \
|
||||
|| fail "PROBE_TARGET must remain /usr/local/sbin/hyb-oa-public-app-health"
|
||||
[[ "$SYSTEMD_DIR" == "/etc/systemd/system" ]] \
|
||||
|| fail "SYSTEMD_DIR must remain /etc/systemd/system"
|
||||
[[ "$SERVICE_NAME" == "hyb-oa-public-app-health" ]] \
|
||||
|| fail "SERVICE_NAME must remain hyb-oa-public-app-health"
|
||||
systemctl disable --now "${SERVICE_NAME}.timer" >/dev/null 2>&1 || true
|
||||
systemctl stop "${SERVICE_NAME}.service" >/dev/null 2>&1 || true
|
||||
rm -f -- "$SYSTEMD_DIR/${SERVICE_NAME}.timer" "$SYSTEMD_DIR/${SERVICE_NAME}.service" "$PROBE_TARGET"
|
||||
systemctl daemon-reload
|
||||
echo "[app-health-installer][PASS] monitor removed; state and shared alert files retained"
|
||||
}
|
||||
|
||||
case "$MODE" in
|
||||
install)
|
||||
install_monitor
|
||||
;;
|
||||
enable)
|
||||
enable_monitor
|
||||
;;
|
||||
uninstall|--uninstall)
|
||||
uninstall_monitor
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: <explicit probe/webhook/helper paths> bash install-offhost-app-health-monitor.sh [install|enable|uninstall]"
|
||||
;;
|
||||
*)
|
||||
fail "unsupported mode"
|
||||
;;
|
||||
esac
|
||||
@@ -340,6 +340,37 @@ RESTORE_SCOPE=shadow \
|
||||
- 确认完整批次已经复制到异机/异盘，而不是仍只保存在应用主机。
|
||||
- 准备独立 smoke 账号，避免使用老板、管理员或真实员工账号跑自动化。
|
||||
|
||||
## 1.6 ARM 公网应用健康监控（启用仍为 EXTERNAL_GATED）
|
||||
|
||||
应用健康监控与自动备份 dead-man 共用受控企微群机器人入口，但使用独立探测器、systemd units、probe state 和 alert state。它从 ARM 每 5 分钟访问公网 HTTPS `/ready`；连续两次失败才告警，一次成功即恢复。不得把它并入备份 receiver 的安装/卸载生命周期，也不得以本地测试通过代替 ARM 已启用证据。
|
||||
|
||||
先在仓库运行无真实消息合同：
|
||||
|
||||
```bash
|
||||
bash scripts/qa/run-app-health-monitor-contract.sh
|
||||
```
|
||||
|
||||
获得 ARM systemd 写入批准后，使用受控配置中的绝对路径安装 disabled units。命令中的 webhook 正文不得作为参数、环境变量或聊天内容出现：
|
||||
|
||||
```bash
|
||||
PROBE_SOURCE=<repo-app-health-probe> \
|
||||
ALERT_WEBHOOK_FILE=<controlled-root-only-webhook-file> \
|
||||
ALERT_HELPER_SOURCE=<repo-group-alert-helper> \
|
||||
ALERT_HELPER_TARGET=<installed-shared-alert-helper> \
|
||||
bash deploy/trialrun/install-offhost-app-health-monitor.sh install
|
||||
```
|
||||
|
||||
安装后先人工执行一次 oneshot 并核对只含稳定状态/类别的输出；另获真实消息与 timer 启用批准后再执行同参数的 `enable`。检查 enabled/active、最近结果和下一次运行：
|
||||
|
||||
```bash
|
||||
systemctl is-enabled hyb-oa-public-app-health.timer
|
||||
systemctl is-active hyb-oa-public-app-health.timer
|
||||
systemctl list-timers hyb-oa-public-app-health.timer --no-pager
|
||||
systemctl status hyb-oa-public-app-health.service --no-pager
|
||||
```
|
||||
|
||||
真实验收需受控制造持续失败，确认首次失败不发、第二次失败只发一条、相同失败降噪、恢复只通知一次。只测试机器人能发送不算健康监控通过。卸载前先记录脱敏状态，随后执行安装器 `uninstall`；该操作只删除应用健康 units 和探测器，保留状态、共用 helper、webhook、备份 receiver、dead-man 和归档。
|
||||
|
||||
## 2. 升级流程（试运行）
|
||||
建议版本化方式：
|
||||
- 每次发布打 tag，例如 `trialrun-2026-02-08-01`
|
||||
|
||||
在新工单中引用
屏蔽一个用户