The Retry Decision a Status Code Can't Make

The platform runs outbound calling campaigns. A tenant uploads a list of contacts (a batch) and the system calls each one with an AI agent, respecting the tenant's concurrency limits and calling windows. The agent places the call over SIP, has the conversation, and when it's done, the AI handler POSTs a webhook back to the outbound service to report what happened.

That webhook is the entire feedback loop. Everything the system knows about how a call went, it learns from that one callback. And the callback has to answer a question that's harder than it looks: should we call this number again?

The naive answer is "look at the status code." The call returned FAILED, so it failed, so retry it. The call returned COMPLETED, so it's done, so don't. This is wrong in both directions, and the reasons it's wrong are the whole substance of this component.

A Code Is a Claim, Not a Conclusion

The AI handler reports a small set of coarse codes: COMPLETED, FAILED, NO_PICKUP, TEMPORARY_UNAVAILABLE, INSUFFICIENT_BALANCE. The problem is that the same code covers situations that should be treated completely differently.

Take FAILED. A call can fail because the SIP session never really established; the carrier dropped it before a word was exchanged. That's infrastructure noise, and the right response is to try again. But a call can also "fail" after a full conversation that simply didn't reach a successful outcome. That's a real result, and retrying it just means calling someone who already heard the pitch and said no.

The status code alone can't distinguish these. What distinguishes them is the corroborating signal riding alongside the code: whether there was any transcription, and how long the call lasted. A FAILED with no transcription and zero duration is a connection that never happened. A FAILED with no transcription but a non-trivial duration is a call that connected but produced no conversation. A COMPLETED with no transcription but a long duration is, paradoxically, a person who picked up and rejected the agent outright: done, not worth another call.

So the first thing the system does with a webhook is not trust the code. It re-derives the outcome from the code plus the evidence.

func FromWebhook(params WebhookParams, minConversationDurationSeconds int64) Classification {
    code := strings.ToUpper(strings.TrimSpace(params.Code))
    if code == "" {
        code = CodeFailed
    }

    switch code {
    case CodeNoPickup:
        return Classification{Code: CodeNoPickup, Retryable: true}
    case CodeFailed:
        if params.EmptyTranscription {
            if params.DurationSeconds <= 0 {
                // Never connected: infrastructure noise. Retry.
                return Classification{Code: CodeFailedSIPException, Retryable: true}
            }
            if params.DurationSeconds < minConversationDurationSeconds {
                // Connected but no real conversation. Retry.
                return Classification{Code: CodeFailedNoConversation, Retryable: true}
            }
        }
        // A genuine failed outcome. Terminal.
        return Classification{Code: CodeFailed, Retryable: false}
    case CodeCompleted:
        if params.EmptyTranscription && params.DurationSeconds > minConversationDurationSeconds {
            // Picked up and rejected. Terminal.
            return Classification{Code: CodeCompletedUserRejection, Retryable: false}
        }
        return Classification{Code: CodeCompleted, Retryable: false}
    case CodeInvalidNumber:
        return Classification{Code: CodeInvalidNumber, Retryable: false}
    case CodeInsufficientBalance:
        return Classification{Code: CodeInsufficientBalance, Retryable: false}
    default:
        return Classification{Code: code, Retryable: false}
    }
}

The output is a normalised code (FAILED_SIP_EXCEPTION, FAILED_NO_CONVERSATION, COMPLETED_USER_REJECTION) that carries more meaning than the coarse code it came from. One ambiguous FAILED fans out into three distinct outcomes, only two of which warrant another attempt. The normalised code is what gets stored, and it's what every downstream decision reads. The raw code from the handler is an input, not a record.

Processing the Webhook Exactly Once Enough

A webhook can arrive more than once. The handler can be retried by the caller, a network hiccup can cause a duplicate POST, or a slow callback can land after the system has already moved on. If processing a webhook twice double-counts an attempt or resurrects a finished contact, the retry logic downstream becomes nonsense.

The guard isn't an idempotency key; it's the contact's own state, read under a row lock. Each contact in a batch has a status (Queued, Calling, Completed, RetryEligible). A webhook is only meaningful for a contact that's currently Calling. The handler locks the row, checks the state, and bails out harmlessly on anything unexpected.

row, err := queries.GetBatchContactNumberForUpdate(ctx, data.GetBatchContactNumberForUpdateParams{
    BatchID:       params.BatchID,
    ContactNumber: params.ContactNumber,
})
if errors.Is(err, pgx.ErrNoRows) {
    return nil // batch was deleted, nothing to do
}

if row.Status != nil && *row.Status != contactstatus.Calling {
    // Already completed, or never dialled. A duplicate or stale webhook. Ignore.
    return nil
}

This is idempotency by state machine rather than by dedup table. A second webhook for the same contact finds it already in Completed and does nothing. The FOR UPDATE lock serialises concurrent callbacks for the same contact, so two webhooks racing each other can't both pass the Calling check. It's cheaper than a dedicated idempotency store and it reuses state the system already has to track anyway.

The rest of the handler runs in that same transaction: classify the outcome, mark the contact Completed with its normalised code, delete the in-flight execution job, and finalise the batch. Either all of it commits or none of it does. There's no window where the contact is marked done but the attempt accounting hasn't caught up.

Retry Is a Batch-Level Round, Not a Per-Call Reflex

Here's the design decision that surprised me most when I worked through it: a retryable outcome does not immediately re-queue the contact. The webhook marks the contact Completed, even the retryable ones. The retry decision happens later, at finalisation, and it operates on the whole batch at once.

When the last active call in a batch wraps up, the finaliser asks: has this batch exhausted its attempt budget? Every batch has a MaxAttempts. If we've used them all, the batch is done regardless of how many contacts would otherwise be retryable. If there's budget left, the finaliser sweeps the batch and promotes the retryable outcomes from Completed to RetryEligible, and the authority for what counts as retryable is a single explicit list.

var retryableCompletedCodes = []string{
    status.Timeout,
    status.TimeoutUnexpectedStatus,
    completedcode.CodeInternalError,
    completedcode.CodeNoPickup,
    completedcode.CodeFailedNoConversation,
    completedcode.CodeFailedSIPException,
}

// Only contacts whose normalised code is in retryableCompletedCodes
// get flipped back to RetryEligible for the next round.
retryUpdate, err := queries.MarkContactsAsRetryEligible(ctx, data.MarkContactsAsRetryEligibleParams{
    BatchID:        batchID,
    CompletedCodes: retryableCompletedCodes,
    // ...
})

This is why the normalisation earlier matters so much. The retry decision isn't made on the coarse FAILED; it's made on the normalised FAILED_SIP_EXCEPTION versus a plain FAILED. The classifier's job was to produce a code precise enough that this list can be a flat set-membership check. All the ambiguity was resolved upstream, so the actual retry gate is dumb and obvious, which is exactly what you want a retry gate to be.

If any retryable contacts were promoted, the finaliser re-arms the batch: it bumps the batch back to InProgress, resets the queue offset, and schedules a fresh init job to dial the RetryEligible contacts in a new round. If nothing is retry-eligible, or the budget is spent, the batch finalises and stops. Retries happen in rounds, bounded by the attempt budget, not as a per-contact scramble the instant a call fails.

Backoff That's Deliberately Boring

Each retry round is delayed, and the delay grows with the number of attempts so a struggling batch backs off instead of hammering. The implementation is intentionally unclever.

func ExponentialWithJitter(attempts int, base time.Duration, cap time.Duration) time.Duration {
    delay := base * time.Duration(1<<attempts)
    jitter := time.Duration(1+rand.Int64N(1000)) * time.Millisecond
    return min(cap, delay+jitter)
}

The delay doubles per attempt, capped at a ceiling, with up to a second of random jitter added so that many batches retrying at once don't realign onto the same tick. It's not "full jitter" or any of the more elaborate schemes; it's exponential growth plus a small random nudge, and that's enough at this scale. The same primitive is reused everywhere a retry delay is needed: the batch initialiser, the executor, the outbox relay, the durable consumers. One boring function, four call sites, no per-component reinvention.

I mention how plain it is on purpose. The temptation with backoff is to reach for the most sophisticated jitter strategy you've read about. But the jitter only has to break synchronisation between independently-failing batches, and a 0–1000ms random offset does that. The exponential part does the actual backing-off. Anything fancier would be solving a problem this system doesn't have.

What the Component Is Really Doing

Described from the outside, this is "the thing that handles call-result webhooks." That framing hides the actual work, which is a small pipeline of judgement: take a coarse, slightly untrustworthy signal from an external system; corroborate it against the evidence that came with it; normalise it into an outcome the platform genuinely understands; record it idempotently against a state machine; and let a budget-bounded finaliser decide whether the campaign tries again.

Each stage exists because the stage before it isn't enough. The code isn't enough, so we corroborate. The corroboration produces an outcome, so we normalise. The webhook can repeat, so we guard with state. The retryable outcomes accumulate, so we batch them into bounded rounds. Pull any stage out and the failure is specific: trust the raw code and you re-dial people who already said no; skip the state guard and a duplicate webhook double-counts an attempt; drop the attempt budget and a bad batch dials forever.

The most interesting decisions in a calling platform aren't in placing the call. They're in interpreting what the call told you, and being honest that a status code, by itself, never tells you enough.