> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lettr.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Retries

> How Lettr retries failed webhook deliveries with exponential backoff across 7 attempts spanning roughly 35 hours.

Lettr automatically retries failed webhook deliveries to ensure you receive all events. Understanding the retry behavior helps you build resilient integrations that handle temporary failures gracefully.

## Retry Schedule

When a webhook delivery fails, Lettr retries with exponential backoff. This approach balances timely delivery with giving your system time to recover from outages.

| Attempt | Delay      | Cumulative Time |
| ------- | ---------- | --------------- |
| 1       | Immediate  | 0               |
| 2       | 1 minute   | 1 minute        |
| 3       | 5 minutes  | 6 minutes       |
| 4       | 30 minutes | 36 minutes      |
| 5       | 2 hours    | 2.6 hours       |
| 6       | 8 hours    | 10.6 hours      |
| 7       | 24 hours   | 34.6 hours      |

After 7 failed attempts spanning approximately 35 hours, the webhook delivery is marked as permanently failed. At this point, no further automatic retries occur.

<Note>
  The retry schedule provides a balance between timely delivery and avoiding overwhelming a struggling endpoint. Most transient issues resolve within the first few retry attempts.
</Note>

## Success Criteria

For a webhook delivery to be considered successful, your endpoint must:

1. **Return a 2xx status code** (200, 201, 202, 204, etc.)
2. **Respond within 30 seconds**

```javascript theme={null}
// Good - Returns 200 immediately
app.post('/webhooks/lettr', (req, res) => {
  res.sendStatus(200);
  // Process asynchronously after responding
  setImmediate(() => processEvent(req.body));
});

// Bad - Processes before responding (may timeout)
app.post('/webhooks/lettr', async (req, res) => {
  await processEvent(req.body); // This might take > 30 seconds
  res.sendStatus(200);
});
```

## Failure Conditions

Webhooks are retried when any of these conditions occur:

| Condition             | Description                              | Retry?                     |
| --------------------- | ---------------------------------------- | -------------------------- |
| HTTP 4xx (except 410) | Client errors (400, 401, 403, 404, etc.) | Yes                        |
| HTTP 5xx              | Server errors (500, 502, 503, etc.)      | Yes                        |
| HTTP 410 Gone         | Indicates endpoint is permanently gone   | **No** - webhook disabled  |
| Connection timeout    | No response within 30 seconds            | Yes                        |
| Connection refused    | Server not accepting connections         | Yes                        |
| DNS failure           | Domain cannot be resolved                | Yes                        |
| SSL/TLS error         | Certificate or handshake issues          | Yes                        |
| HTTP 2xx              | Success responses                        | **No** - delivery complete |

### The 410 Gone Response

Returning a `410 Gone` status code is a signal to permanently disable the webhook. Use this when:

* You're decommissioning an endpoint
* The webhook should no longer receive events
* You want to stop retries without deleting the webhook via API

```javascript theme={null}
// Permanently disable this webhook
app.post('/webhooks/lettr', (req, res) => {
  if (shouldDisableWebhook()) {
    return res.sendStatus(410); // Webhook will be disabled
  }
  // Normal processing
  res.sendStatus(200);
});
```

## Idempotency and Duplicate Handling

Because webhooks can be retried, your endpoint may receive the same event multiple times. Always implement idempotent handling.

### Why Duplicates Occur

1. **Network issues**: Your server responds 200, but the response doesn't reach Lettr
2. **Timeout at boundary**: Processing completes at exactly 30 seconds
3. **Infrastructure retries**: Load balancers or proxies may retry requests

### Handling Duplicates

Use the event `id` to detect and skip duplicate deliveries:

```javascript theme={null}
import { Redis } from 'ioredis';

const redis = new Redis(process.env.REDIS_URL);
const PROCESSED_EVENT_TTL = 86400; // 24 hours

app.post('/webhooks/lettr', async (req, res) => {
  const event = req.body;

  // Check if already processed
  const key = `webhook:${event.id}`;
  const alreadyProcessed = await redis.get(key);

  if (alreadyProcessed) {
    console.log(`Duplicate event ${event.id}, skipping`);
    return res.sendStatus(200); // Still return 200!
  }

  // Mark as processed (with TTL)
  await redis.set(key, '1', 'EX', PROCESSED_EVENT_TTL);

  // Process the event
  await handleEvent(event);

  res.sendStatus(200);
});
```

<Warning>
  Always return a 200 status code for duplicate events. Returning an error will trigger unnecessary retries.
</Warning>

### Database-Based Deduplication

If you don't have Redis, use your database:

```javascript theme={null}
app.post('/webhooks/lettr', async (req, res) => {
  const event = req.body;

  try {
    // Insert with unique constraint on event_id
    await db.processedWebhooks.insert({
      event_id: event.id,
      event_type: event.type,
      received_at: new Date()
    });
  } catch (err) {
    if (err.code === '23505') { // PostgreSQL unique violation
      console.log(`Duplicate event ${event.id}, skipping`);
      return res.sendStatus(200);
    }
    throw err;
  }

  // Process the event
  await handleEvent(event);
  res.sendStatus(200);
});
```

## Monitoring Webhook Health

### Webhook Status

Each webhook exposes two status fields via the API:

| Field         | Values                             | Description                                                                                                  |
| ------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `enabled`     | `true` / `false`                   | Whether the webhook is currently enabled (can be toggled manually or auto-disabled after sustained failures) |
| `last_status` | `"success"` / `"failure"` / `null` | The result of the most recent delivery attempt (`null` if no attempts yet)                                   |

### Check Webhook Status via API

You can check webhook status using the read-only API:

```bash theme={null}
curl -X GET "https://app.lettr.com/api/webhooks/{webhookId}" \
  -H "Authorization: Bearer lttr_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```

The response includes status information:

```json theme={null}
{
  "message": "Webhook retrieved successfully.",
  "data": {
    "id": "webhook-abc123",
    "name": "Order Notifications",
    "url": "https://example.com/webhook",
    "enabled": true,
    "event_types": ["message.delivery", "message.bounce"],
    "auth_type": "basic",
    "has_auth_credentials": true,
    "last_successful_at": "2024-01-15T10:30:00+00:00",
    "last_failure_at": null,
    "last_status": "success"
  }
}
```

### Dashboard Monitoring

The Lettr dashboard provides visibility into webhook delivery:

1. Navigate to **Webhooks** in the sidebar
2. Select a webhook to view its details
3. See last attempt time, last status, and enabled state

## Automatic Disabling

Webhooks are automatically disabled after sustained failures to protect both systems:

* Prevents queue buildup of undeliverable events
* Reduces load on your failing endpoint
* Alerts you to investigate the issue

When a webhook is auto-disabled, you'll see it in the dashboard and can re-enable it after fixing the issue. Navigate to **Webhooks** in the sidebar, select the disabled webhook, and toggle it back on.

<Warning>
  Before re-enabling, ensure the underlying issue is resolved. Re-enabling without fixing the problem will lead to immediate failures and potentially another auto-disable.
</Warning>

## Best Practices for Reliable Delivery

### 1. Respond Quickly

Return a 200 response as fast as possible. Process events asynchronously:

```javascript theme={null}
app.post('/webhooks/lettr', (req, res) => {
  // Acknowledge immediately
  res.sendStatus(200);

  // Process in background
  setImmediate(async () => {
    try {
      await processEvent(req.body);
    } catch (err) {
      console.error('Processing failed:', err);
      // Store for manual retry or alerting
      await storeFailedEvent(req.body, err);
    }
  });
});
```

### 2. Use a Queue

For high-volume or complex processing, use a message queue:

```javascript theme={null}
import { Queue } from 'bullmq';

const webhookQueue = new Queue('webhooks');

app.post('/webhooks/lettr', async (req, res) => {
  // Add to queue with deduplication
  await webhookQueue.add(req.body.type, req.body, {
    jobId: req.body.id // Prevents duplicate jobs
  });

  res.sendStatus(200);
});
```

### 3. Handle Partial Failures

If processing involves multiple steps, handle partial failures gracefully:

```javascript theme={null}
async function processEvent(event) {
  // Critical operation - must succeed
  await updateDatabase(event);

  // Non-critical operations - fail gracefully
  const nonCriticalTasks = [
    sendNotification(event).catch(err => {
      console.warn('Notification failed:', err);
    }),
    updateAnalytics(event).catch(err => {
      console.warn('Analytics failed:', err);
    })
  ];

  await Promise.all(nonCriticalTasks);
}
```

### 4. Monitor and Alert

Set up monitoring for webhook health:

```javascript theme={null}
// Track webhook processing metrics
const metrics = {
  received: 0,
  processed: 0,
  failed: 0,
  duplicates: 0
};

app.post('/webhooks/lettr', async (req, res) => {
  metrics.received++;

  const isDuplicate = await checkDuplicate(req.body.id);
  if (isDuplicate) {
    metrics.duplicates++;
    return res.sendStatus(200);
  }

  try {
    await processEvent(req.body);
    metrics.processed++;
  } catch (err) {
    metrics.failed++;
    console.error('Webhook processing failed:', err);
    // Alert if failure rate is high
    if (metrics.failed / metrics.received > 0.1) {
      await alertOps('High webhook failure rate');
    }
  }

  res.sendStatus(200);
});
```

### 5. Implement Health Checks

Ensure your webhook endpoint is monitored:

```javascript theme={null}
// Health check endpoint
app.get('/webhooks/health', (req, res) => {
  const healthy = checkDatabaseConnection() && checkQueueConnection();
  res.status(healthy ? 200 : 503).json({ healthy });
});
```

## Troubleshooting

### Common Issues

| Problem                 | Possible Cause                  | Solution                                |
| ----------------------- | ------------------------------- | --------------------------------------- |
| All webhooks timing out | Slow processing before response | Return 200 immediately, process async   |
| Intermittent failures   | Resource exhaustion             | Add queue, increase capacity            |
| SSL errors              | Certificate issues              | Verify certificate chain, check expiry  |
| 4xx errors              | Authentication/authorization    | Check auth config, verify endpoint path |
| No webhooks received    | Webhook disabled                | Check status in dashboard, re-enable    |

### Debugging Failed Deliveries

1. **Check delivery history** in dashboard or via API
2. **Review response codes** and error messages
3. **Check your server logs** for the corresponding requests
4. **Verify endpoint URL** is correct and accessible
5. **Test with manual retry** after fixing issues

## Event Ordering

Webhooks are delivered in approximate order, but strict ordering is not guaranteed. Events may arrive out of order due to:

* Retry delays
* Network latency variations
* Parallel processing

Design your handlers to be order-independent when possible:

```javascript theme={null}
async function handleEmailStatus(event) {
  const { emailId, status, timestamp } = event.data;

  // Use timestamp to handle out-of-order updates
  await db.emails.update(
    { id: emailId },
    {
      status,
      status_updated_at: timestamp
    },
    {
      // Only update if this event is newer
      where: {
        OR: [
          { status_updated_at: null },
          { status_updated_at: { lt: timestamp } }
        ]
      }
    }
  );
}
```

## Related Topics

<CardGroup cols={2}>
  <Card title="Handling Webhooks" icon="code" href="/learn/webhooks/handling">
    Best practices for processing webhooks reliably
  </Card>

  <Card title="Authorization" icon="lock" href="/learn/webhooks/authorization">
    Secure your endpoints with authentication
  </Card>

  <Card title="Event Types" icon="list" href="/learn/webhooks/event-types">
    Complete reference of all webhook events
  </Card>

  <Card title="Testing" icon="flask" href="/learn/webhooks/testing">
    Test webhooks in development and production
  </Card>
</CardGroup>