Webhooks look like the simplest integration available: an event happens, you post it to a URL. Both ends turn out to be harder than that, and the characteristic failure is silence - the sender believes it delivered, the receiver never processed anything, and nobody notices until someone asks why an order never appeared. Building reliable webhooks is mostly a matter of accepting that delivery is uncertain and designing both sides accordingly.
If you are sending, deliver at least once and say so in your documentation. Networks fail, receivers deploy, and a response that never arrives does not mean the work was not done. Retry with exponential backoff over a window long enough to survive a receiver's deployment or brief outage, then stop into a dead letter store a human can inspect and replay. A sender that tries three times over a minute and gives up silently loses events during every routine restart at the other end.
Sign what you send. The receiving endpoint is public, so anything on the internet can post to it, and a receiver that trusts the payload is trusting strangers. A signature over the raw body using a shared secret, with a timestamp inside the signed material to bound replay, is the standard approach and is not difficult to implement. Publish how to verify it, and allow customers to rotate the secret without downtime, which means supporting two valid secrets during the overlap.
If you are receiving, do almost nothing inside the handler. Validate the signature, write the payload to a queue or a table, and return success. Processing inline makes your handler's duration the sender's timeout budget, and any slow downstream call becomes a delivery failure and a retry - which arrives while the first attempt is still running. Acknowledging quickly and processing asynchronously removes an entire category of duplicated work.
Assume duplicates and assume disorder. At-least-once delivery means the same event will arrive twice, and retries mean an older event can land after a newer one. Receivers need an event identifier to deduplicate on, and state transitions that tolerate arriving out of sequence, which usually means acting on the state carried in the event rather than assuming the previous one was applied. Systems built on the assumption of ordered, exactly-once delivery work until the first retry storm.
Make failure visible at both ends. Senders should expose recent delivery attempts and the responses received, so an integrator can diagnose without opening a support ticket. Receivers should alert when the rate of incoming events falls to nothing, because zero is a plausible-looking value that hides a broken endpoint. The absence of events is the hardest failure to notice and the one most often discovered late.
Where webhooks are the wrong tool, use polling and stop apologising for it. If the receiver cannot expose a public endpoint, if event volume is very high, or if the consumer needs to control when it does work, a paged API read on the consumer's own schedule is simpler and far easier to debug. Offering both, and letting the integrator choose, costs less than forcing one mechanism to do everything.
