volcengine Follow this procedure. It exists because the obvious answer is usually wrong.
1. Pull everything first
Fetch logs, metrics and deploy history for the whole window before you analyze any of them. A conclusion drawn from one source is a guess.
2. Characterize signatures, do not rank them
Never conclude from "which error is most frequent". High-volume errors are usually chronic noise that was there yesterday too.
For every distinct error signature, compute its count per hour across the window. You are looking for a signature whose rate changed — ideally one that was zero and then was not. A signature that is flat across the whole window is background, however loud it is.
3. Locate the change point
For each signature that changed, find the timestamp of its first occurrence after the change. That timestamp, not the start of the window, is the moment you are explaining.
4. Correlate with deploys
Normalize every timestamp to UTC epoch seconds before comparing sources; they do not all use the same format or the same clock. A deploy is a candidate only if it precedes the change point by minutes, not hours. Where two deploys are close together, the one that lines up is the one that lines up — check both.
5. Confirm in the metrics
The metric series is long format: one row per metric per minute. Aggregate it per metric before drawing conclusions.
- A cause produces a sustained change beginning at the change point. A spike that recovers on its own is not your incident.
- Check request volume too. If traffic did not change, the incident is not load-driven and you should stop looking for one.
- A resource metric sitting exactly at its configured limit is worth more than any latency graph.
6. Explain the mechanism
Tie the candidate deploy's change list to what the metrics show. If you cannot explain how that change produces these numbers, you have a correlation, not a root cause — and you should say that in the ticket.
7. File one ticket
Exactly one, at the end. Every claim in evidence must be a number you
actually computed, with its timestamp. Rule out the hypotheses you rejected,
and say why.
Working notes
Keep your analysis scripts in the workspace under analysis/ with names that
say what they do. You will be asked to re-run them over a different window.
Take the input file path as a command-line argument (sys.argv[1]). Never
hardcode it — a script with a filename baked in is not reusable, and renaming
data files to fit an old script wastes far more time than adding one argument.