Raz-Gits integration forensics
When your code talks to a system you do not control, the bug is usually not in your code. It is a false belief about what the other side does, and no amount of reading your own code will surface it.
So the method inverts. Do not reason about the code. Establish what actually crossed the wire, then reason about the difference between that and what you expected.
The documentation describes intended behaviour. You need observed behaviour, and they are different systems.
Step 1, get the raw payload, before anything touches it
Not the parsed object. Not the logged summary. The bytes.
Most integration bugs are invisible after parsing, because the parse is where the wrong assumption gets applied. A field arrives as a string instead of a number, an array has one element instead of five, a key is absent rather than null, and the parsed object looks perfectly reasonable.
If there is no raw log, adding one is the first fix, not a detour. Log the untouched body and the headers, on every inbound call, before validation. Keep a rolling window. This one change resolves more integration bugs than any amount of instrumentation downstream, and it costs almost nothing.
Then reproduce from a stored payload, never from live traffic. Live traffic is not reproducible, arrives when it wants, and cannot be replayed against a fix. A saved payload is a test case.
Step 2, the catalogue
Nearly every third-party integration bug is one of these. Work down the list against the raw payload rather than guessing.
It returns success for something that is gone. A 200 with a body, for a resource that no longer exists. Job boards do this constantly: the posting page serves fine while the job is absent from the board's own listing API. Never treat a status code as proof a thing exists. Check it against the collection it should be in.
One logical action fires many events. A phone call is not one webhook. It is Setup, Proceeding, Answered, Disconnected, each a separate delivery. If your handler assumes one event per call, everything downstream double-counts. Find out how many events the vendor emits per real-world action before writing the handler.
Events go missing. The provider drops one, usually the terminal one, and now you have a record that never closes. Assume the closing event may never arrive and build the sweep that closes stale records on its own. This is not a rare edge; it is a weekly occurrence at volume.
Events arrive out of order. The completion lands before the creation. Ordering is not guaranteed unless the vendor explicitly guarantees it, and mostly they do not. Handle by state rather than by sequence.
Retries look like new events. The vendor did not get your 200 fast enough and sent it again. Without an idempotency key on the event id, you process it twice. Check whether their retry carries the same id or a new one, because both designs exist.
Documented fields are absent, or a different type. Especially on older records, free accounts, and objects created before their last migration. Never let a missing optional field crash the whole batch.
Success that silently truncates. Rate limits and pagination that return 200 with fewer results rather than an error. If a count looks round, or exactly at a limit, suspect this before suspecting your filter.
A field that contradicts the record. The metadata says one thing and the actual object says another. A listing marked remote whose own application form requires five days on site. When a summary field and the detail disagree, the detail is the truth and the summary is marketing.
Step 3, bisect the boundary
Once you have the raw payload, the question is only ever which side is wrong, and there are three places to look:
- What they sent. Compare the raw body to their docs. A mismatch here is their bug, or your version is out of date, and either way the fix is not in your parser.
- What you understood. Parse the raw body in isolation. If the parse is wrong, you have found it, and now you have a regression test for free.
- What you did with it. Only reach here after the first two are cleared.
Doing these out of order is what turns an afternoon into a week.
Step 4, build the thing that would have told you
An integration that broke silently will break silently again. Before closing it out, add whichever of these is missing:
- A heartbeat. Write a timestamp on every successful run and alert when it goes stale. This catches the worst failure mode, which is a job that exits zero and writes nothing. A non-zero exit gets noticed; silence does not.
- A floor. If the normal result is thousands of rows, zero is a failure, not a quiet day. Assert it.
- A comparison to yesterday. A collapse from four thousand to six is not a crash and no exception will fire.
- The raw log, if you had to add it in step 1. Keep it.
When their system and their docs disagree
The observed behaviour is the specification. Build for what you measured, leave a comment saying it contradicts the documentation and on what date you measured it, and move on. Waiting for a vendor to fix their docs is not a plan.
Report it to them anyway. It occasionally works, and it costs one email.
Rules
- Never theorise before reading the payload. The whole failure mode of integration debugging is confident reasoning about imagined data.
- One change at a time. Fixing three suspected causes at once means never learning which one it was, and integration bugs recur.
- Their status code is not evidence. Neither is their summary field.
- Write the regression test from the real payload, exactly as received, including the parts you think are irrelevant.
- Say "I don't know what they send" out loud rather than assuming. It is the sentence that leads to looking, and looking is the entire method.