Byted Util Volcengine Detect Retry

Volcengine Detect Retry

Reduce Volcengine Cloud Detect (云拨测) operations toil. This skill confirms whether a failing dial point is a real outage or just noise, and — when a dial point itself is broken — suggests a similar replacement node.

This skill ships no script of its own. It is pure orchestration: every action is one of the byted-util-volcengine-cloud-detect skill's volc_detect.py CLI commands (get-task, get-result, list-nodes, fuzzy-search-nodes, create-task, delete-task). Follow the workflow below by invoking those commands in sequence and applying the decision logic to their JSON output.

Requires: byted-util-volcengine-cloud-detect skill. Do not reimplement any Cloud Detect API call here.

Locate The Base CLI

Resolve volc_detect.py dynamically (never hard-code an absolute path):

VOLC_DETECT="${VOLC_DETECT:-$(find "${SKILLS_DIR:-$HOME/Code/Skill}" -type f \
  -path '*/byted-util-volcengine-cloud-detect/scripts/volc_detect.py' 2>/dev/null | head -n 1)}"
test -n "$VOLC_DETECT" || { echo "volc_detect.py not found"; exit 1; }
alias vd="python3 \"$VOLC_DETECT\""

All commands below are written as python3 "$VOLC_DETECT" <command>.

Prerequisites

Volcengine credentials must be present (the base skill enforces this). If either is missing, follow the base skill's Prerequisites section to collect and persist them:

test -n "$VOLC_ACCESSKEY" && test -n "$VOLC_SECRETKEY"

Decision Logic

For each dial point whose latest sample in the scan window failed:

1. Retry 5 times (configurable) against the same monitored target via a temporary create-task.
   • All retries succeed → transient glitch → ignore (transient_issue).
   • Any retry fails → go to control validation.
2. Control validation: a temporary create-task dials a trusted domain from the same node.
   • China nodes → https://cn.bing.com
   • Overseas nodes → https://www.google.com
   • Control succeeds → real_failure (target is genuinely down for that node) → ALERT.
   • Control fails → node_issue (the dial point itself is broken) → find similar replacement nodes and suggest replacing it.
3. Temporary retry/control tasks are always deleted (delete-task).

By default only LastMile dial points (client_info.type == "1", node name suffix （LM）) are auto-validated, because they are the noisy ones. Include IDC/Private only on request.

Replacement-node priority (node_issue only):

1. Mandatory: same node type (LastMile / IDC / Private).
2. Preferred: same province, then same country.

Success Rule

A result sample is success when basic_detail.usability_info.status is "success" (or boolean true); it is a failure when status is "fail" (the usability_info.reason array, e.g. ["主机未找到(601)"], and basic_detail.error_msg explain why). If no usability_info is present, fall back to http_detail.http_code in 200–499 = reachable.

Workflow: Scan One Periodic Task

Step 1 — Fetch the task and its recent samples

# Task must be Running (status == 2) to be actively monitoring.
python3 "$VOLC_DETECT" get-task --task-id <TASK_ID>

# Pull the last hour of results (API limits the range to <= 1 hour, PageSize <= 500).
NOW=$(date +%s); START=$((NOW-3600))
python3 "$VOLC_DETECT" get-result --task-id <TASK_ID> \
  --start-time $START --end-time $NOW --page-size 500

Step 2 — Identify failing nodes (latest sample per node)

From the Data array, group samples by node and keep the latest (max basic_detail.timestamp) per node. A node is identified by its basic_detail.client_info (region / city / isp / type). Map that back to a node id + full name with list-nodes (match region, city, isp substrings; if client_info.type is present, also match node type to disambiguate LM vs IDC).

python3 "$VOLC_DETECT" list-nodes

Keep only nodes whose latest sample is a failure. By default, drop non-LastMile nodes (client_info.type != "1") unless the user asked to include IDC/Private. If a client_info maps to multiple nodes (ambiguous) or none, mark it inconclusive and do not auto-page.

Step 3 — Retry each failing node 5× against the same target

get-task gives the monitored address. Create a temporary task on the failing node only:

ADDRESS="<the task's address>"; LINE_ID=<failing node id>
NOW=$(date +%s); FINISH=$((NOW + 5*60 + 120))   # 5 rounds @ 60s + margin
python3 "$VOLC_DETECT" create-task \
  --address "$ADDRESS" --name "retry_${LINE_ID}_${NOW}" \
  --type 1 --node-count 1 --interval-seconds 60 \
  --finish-time $FINISH --line-ids $LINE_ID --http-method 1

Wait for ~5 samples (poll get-task until status == 6, or sleep rounds × interval), fetch the results, then delete the temporary task:

END=$(date +%s); START=$((END-3600))
python3 "$VOLC_DETECT" get-result --task-id <RETRY_TASK_ID> \
  --start-time $START --end-time $END --page-size 500
python3 "$VOLC_DETECT" delete-task --task-id <RETRY_TASK_ID>

• All retry samples succeed → transient_issue → ignore. Done for this node.
• Any retry fails → continue to Step 4.

Step 4 — Control validation from the same node

Pick the control domain by node locality (node name starting with 中国 → CN):

# China node:   CONTROL="https://cn.bing.com"
# Overseas node: CONTROL="https://www.google.com"
NOW=$(date +%s); FINISH=$((NOW + 90))
python3 "$VOLC_DETECT" create-task \
  --address "$CONTROL" --name "control_${LINE_ID}_${NOW}" \
  --type 1 --node-count 1 --interval-seconds 60 \
  --finish-time $FINISH --line-ids $LINE_ID --http-method 1

Wait for a sample, fetch results, then delete the control task:

END=$(date +%s); START=$((END-3600))
python3 "$VOLC_DETECT" get-result --task-id <CONTROL_TASK_ID> \
  --start-time $START --end-time $END --page-size 50
python3 "$VOLC_DETECT" delete-task --task-id <CONTROL_TASK_ID>

• Control succeeds → real_failure: the monitored target is genuinely down for this node while the node itself is healthy → ALERT.
• Control fails → node_issue: the dial point itself is broken → go to Step 5.
• No control samples → inconclusive (do not auto-page).

Step 5 — Suggest replacement nodes (node_issue only)

Find same-type, preferably same-province (then same-country) candidates, excluding the broken node. fuzzy-search-nodes filters by name substrings (国家-省份-城市-运营商（类型）):

# e.g. broken node = 中国-河南-三门峡市-中国电信（LM）
python3 "$VOLC_DETECT" fuzzy-search-nodes "河南 电信 LM"   # same province first
python3 "$VOLC_DETECT" fuzzy-search-nodes "中国 电信 LM"   # fall back to same country

Present the top candidates and recommend swapping the node in the source periodic task's LineIdList.

Classification Reference

Reporting

Summarize the scan as a single object the operator can act on:

{
  "alert": true,
  "message": "1 confirmed real failure(s), 1 broken node(s), 2 transient (ignored), 0 inconclusive",
  "real_failures": [
    {
      "task_id": "23275831",
      "line_id": 823,
      "address": "https://example.com",
      "node_name": "中国-河南-三门峡市-中国电信（LM）",
      "classification": "real_failure",
      "control_address": "https://cn.bing.com",
      "details": "Retries failed but the control target succeeded from the same node."
    }
  ],
  "node_issues": [
    {
      "line_id": 229,
      "node_name": "中国-河北-石家庄市-中国电信（LM）",
      "classification": "node_issue",
      "control_address": "https://cn.bing.com",
      "suggested_replacements": [
        {"line_id": 380, "node_name": "中国-河南-南阳市-中国电信（LM）", "match": ["same_type", "same_province", "same_country"]}
      ],
      "details": "Both the monitored target and the control target failed from this node."
    }
  ],
  "suppressed": [{"line_id": 230, "classification": "transient_issue"}],
  "inconclusive": []
}

How To Act On The Output

• real_failures → page / alert. The monitored site is genuinely down for those nodes.
• node_issues → the dial point is broken. Present suggested_replacements and recommend swapping the node in the periodic task's LineIdList.
• suppressed (transient) → no action; log for auditability.
• inconclusive → insufficient evidence; surface in the report but do not auto-page.

alert is true only when there is at least one real_failure.

Notification

If the user provides a webhook or notify script, send only the real_failures set as the paging payload, and include node_issues (with replacement suggestions) so operators can act. Include suppressed/inconclusive counts in the text for auditability.

Scheduling (hourly)

Run the scan workflow on whatever scheduler you use (cron / CI). The 1-hour scan window matches an hourly cadence; shrink the window if you run more frequently.

Resources

• byted-util-volcengine-cloud-detect skill: provides volc_detect.py. This skill calls its commands directly and adds only the retry → control → replacement decision logic above.