Selfhosted Gateway Seamless Upgrade

Selfhosted Gateway Seamless Upgrade

Use this skill for the current StaticFlow production shape:

• stable gateway: 127.0.0.1:39180
• backend slots: blue -> 39080, green -> 39081
• gateway config: conf/pingora/staticflow-gateway.yaml
• process supervision: usually tmux, not systemd

Non-Negotiable Rules

• Do not stop, kill, restart, or recreate the gateway process during backend rollout.
• Do not touch the gateway to fix tmux session names, process ancestry, or operational neatness.
• Do not inspect or use gateway lifecycle helper scripts as the rollout plan. In tmux production they are a distraction; the rollout path is the manual sequence in this skill.
• The only allowed gateway signal during cutover is: kill -HUP <gateway-pid>.
• If an action would interrupt 39180, stop and ask the user.
• Direct checks on 39080 or 39081 only prove the candidate slot works. Cutover is proven only through 39180 and, when public production matters, https://ackingliu.top.
• Keep the old slot alive unless repeated stable-path checks prove the new slot is serving real traffic.

Forbidden during normal hot update:

• tmux kill-session -t sf-gateway
• kill <gateway-pid> without -HUP
• systemctl restart staticflow-gateway...
• any gateway stop/start cycle
• switching traffic because a candidate slot alone is healthy

Required Preflight

Before building or starting anything, identify the live topology with read-only commands:

curl -fsS http://127.0.0.1:39180/api/healthz
env -u https_proxy -u HTTPS_PROXY -u http_proxy -u HTTP_PROXY -u all_proxy -u ALL_PROXY \
  curl -fsS https://ackingliu.top/api/healthz
ss -ltnp | grep -E ':(39180|39080|39081)\b'
tmux ls
tmux list-panes -a -F '#{session_name}:#{window_index}.#{pane_index} pid=#{pane_pid} cmd=#{pane_current_command}'
awk '/active_upstream:/ {print $2; exit}' conf/pingora/staticflow-gateway.yaml

Confirm:

• local 39180 health and public health report the same active backend port
• the reported active port matches active_upstream in the config
• the other slot port is free before starting the candidate
• the gateway pid is the listener on 39180

If public health does not go through 39180 to the same active slot, this skill does not apply. Stop and ask.

Build Rule

Only run one Rust build/check at a time. When a live backend is running, cap Cargo parallelism:

CARGO_BUILD_JOBS=1 cargo build --profile release-backend -p static-flow-backend --jobs 1

Start the candidate from:

target/release-backend/static-flow-backend

Do not overwrite bin/static-flow-backend before cutover.

Candidate Slot

Choose the inactive slot:

• if active is green, candidate is blue on 39080
• if active is blue, candidate is green on 39081

Start the candidate in tmux with the freshly built binary. Use the existing session naming convention if one is already present; common names are sf-backend-blue and sf-backend-green.

Example for blue:

tmux new-session -d -s sf-backend-blue \
  'cd /home/ts_user/rust_pro/static_flow && BACKEND_BIN=/home/ts_user/rust_pro/static_flow/target/release-backend/static-flow-backend DB_ROOT=/mnt/wsl/data4tb/static-flow-data PORT=39080 STATICFLOW_LOG_SERVICE=backend-blue-39080 ./scripts/start_backend_selfhosted.sh --port 39080'

Example for green:

tmux new-session -d -s sf-backend-green \
  'cd /home/ts_user/rust_pro/static_flow && BACKEND_BIN=/home/ts_user/rust_pro/static_flow/target/release-backend/static-flow-backend DB_ROOT=/mnt/wsl/data4tb/static-flow-data PORT=39081 STATICFLOW_LOG_SERVICE=backend-green-39081 ./scripts/start_backend_selfhosted.sh --port 39081'

Candidate checks before cutover:

curl -fsS http://127.0.0.1:<candidate-port>/api/healthz
curl -fsS 'http://127.0.0.1:<candidate-port>/api/articles?limit=1'
curl -fsS 'http://127.0.0.1:<candidate-port>/api/llm-gateway/status'

For feature-specific changes, add the exact endpoint that proves the fix.

Cutover

Cutover is config edit plus in-process reload.

1. Back up the config:

   cp conf/pingora/staticflow-gateway.yaml tmp/staticflow-gateway.$(date +%Y%m%d-%H%M%S).yaml.bak
   

2. Change only the active_upstream line to the candidate slot.

3. Re-read the config and confirm it now names the candidate:

   awk '/active_upstream:/ {print $2; exit}' conf/pingora/staticflow-gateway.yaml
   

4. Find the running gateway pid from the 39180 listener:

   ss -ltnp | grep ':39180 '
   

5. Reload the already-running gateway:

   kill -HUP <gateway-pid>
   

Do not replace step 5 with restart.

Stable-Path Verification

After SIGHUP, verify through the stable path. Repeat until local and public health pass at least 3 consecutive times and each response reports the candidate port.

for i in 1 2 3; do
  curl -fsS http://127.0.0.1:39180/api/healthz
  curl -fsS 'http://127.0.0.1:39180/api/articles?limit=1'
  curl -fsS 'http://127.0.0.1:39180/api/llm-gateway/status'
  env -u https_proxy -u HTTPS_PROXY -u http_proxy -u HTTP_PROXY -u all_proxy -u ALL_PROXY \
    curl -fsS https://ackingliu.top/api/healthz
  sleep 1
done

Only after these stable-path checks pass should you say the rollout is live. If they do not stay on the candidate port, keep the old slot alive and roll back.

Rollback

Rollback is restore config plus SIGHUP.

1. Restore the backed-up gateway config.
2. Confirm active_upstream names the old slot.
3. Send kill -HUP <gateway-pid> to the same running gateway process.
4. Verify http://127.0.0.1:39180/api/healthz reports the old slot port.

Do not restart the gateway to roll back.

Cleanup

Cleanup is optional and happens after cutover proof, not during cutover.

• Stop only the old backend slot supervisor, never the gateway.

• Stop the old slot only after repeated stable-path checks prove traffic is on the new slot.

• If any check is ambiguous, leave the old slot running.

• After cleanup, confirm only the old backend port stopped listening.

• Then, if needed, sync the release artifact with a new inode:

  install -m 755 target/release-backend/static-flow-backend bin/static-flow-backend.new
  mv -f bin/static-flow-backend.new bin/static-flow-backend
  

Common Distraction Traps

• A gateway process not being under the expected tmux session name is not a rollout problem. Do not restart it.
• A helper script exposing restart, reload, or switch subcommands is not permission to use them in tmux production.
• ps can show stale tmux server argv mentioning an old session. Treat tmux ls, tmux list-panes, and ss -ltnp as the runtime truth.
• A candidate slot being healthy does not authorize stopping the old slot.
• A single successful 39180 response is not enough for cleanup.

Stop Conditions

Stop and ask the user if:

• gateway pid cannot be identified from the 39180 listener
• local and public health disagree about the active slot
• the inactive slot is not actually free
• the candidate does not pass direct health and feature checks
• SIGHUP does not move 39180 to the candidate slot
• stable-path verification is not repeatedly successful
• any proposed action would stop, restart, or recreate the gateway process