Zero-downtime upgrades for a self-hosted error tracker.

What "zero-downtime" actually means for an error tracker

Zero-downtime in the context of an error tracker is not the same as zero-downtime for a user-facing web application. The SDK is the client. The SDK retries. The distinction matters because it determines how long your upgrade window can be before events start dropping.

SDK retry behavior

Every Sentry SDK ships with a retry and local queue mechanism. When a POST /api/PROJECT_ID/store/ or POST /api/PROJECT_ID/envelope/ request fails with a 5xx status or a connection error, the SDK does not discard the event. It holds the event in a local in-memory buffer and retries after a short backoff, typically two to five seconds. The buffer capacity and discard policy vary by SDK language, but the baseline across the official SDKs is a 30-second hold window before the event is abandoned.

A 503 response with a Retry-After: 5 header tells the SDK to back off for five seconds before retrying. The nginx and Caddy reload semantics described later in this guide produce exactly this behavior: the proxy returns 503 for in-flight requests during the drain window, the SDK retries, and the new binary is ready to serve the retry.

The cost of a 30-second drop

At 400 events per second, a 30-second window where ingest is entirely unavailable and SDKs discard rather than retry means 12,000 dropped events. Most of those events are noise (routine warnings, expected errors, high-frequency low-severity issues). The events that hurt are the high-severity first-occurrence errors that appear precisely during a deploy, when the new code has a bug that the error tracker would catch. Dropping those events removes the signal you need most.

The practical target is a drain window under 10 seconds for a standard urgentry deployment. At that window, no SDK will have exhausted its retry budget, no event is lost to discard, and the gap in event timestamps is small enough that grouping and alert logic remains coherent.

Dropped events vs dropped queries

A dropped event is an event the SDK sent that urgentry never recorded. A dropped query is a UI request (loading the issues list, viewing an event detail) that returned an error because the server was mid-restart. These are different failure modes with different costs.

Dropped queries affect engineers looking at the dashboard during an upgrade. They are annoying. They are not data loss. A 5xx on a UI request means the engineer hits refresh and gets the result two seconds later when the new binary is running.

Dropped events are permanent. The SDK has moved on. urgentry does not replay events from the SDK. An event that was discarded during the SDK retry window is gone. The upgrade procedure in this guide targets zero dropped events, not zero dropped queries.

The two upgrade shapes

urgentry supports two upgrade paths. Choose based on your tolerance for a brief planned drain window and your need for a hard rollback path.

Binary swap with SIGHUP reload

The standard path for single-VPS deployments. The sequence: drain the ingest queue, stop the running binary, start the new binary (migrations run on startup), send SIGHUP to signal readiness to the reverse proxy. Total elapsed time is three to ten seconds depending on migration complexity and queue depth.

This is the path most urgentry operators use. It requires one host, one binary, and a reverse proxy that handles the brief unavailability window with a 503 response. The rollback path is restoring the previous binary file and restarting.

Blue/green behind a reverse proxy

The higher-assurance path. Run two urgentry binaries on two ports. The reverse proxy sends all traffic to the current (blue) instance. When the new (green) instance is ready, shift the proxy weight to 100% green. The blue instance continues to drain in-flight requests for a short period, then stops.

Blue/green eliminates the drain window at the expense of running two instances simultaneously during the cutover. For a deployment on a $5 VPS with 1 GB of RAM, two urgentry instances at 52 MB resident each consume 104 MB, leaving plenty of headroom. The rollback path is shifting the proxy weight back to blue without touching the database.

SDK-side forward compatibility

Sentry SDKs are built to tolerate short server unavailability. Understanding the specifics lets you size your upgrade window with confidence.

What each SDK does on a 503

The Python SDK holds events in a transport queue backed by a thread-safe deque. On a 5xx response, it retries with exponential backoff up to three attempts before logging a warning and discarding. The JavaScript SDK uses fetch with a timeout; on failure it queues the event in localStorage (browser) or an in-memory buffer (Node.js) and retries on the next scheduled flush (every two seconds by default). The Go SDK uses a similar in-memory buffer with configurable flush intervals. The Java and Android SDKs persist events to disk between sessions, which means events survive process restarts on the SDK side.

The important point: a 503 that lasts under 10 seconds will not cause a drop in any of the mainstream SDKs. A 503 that lasts 30 seconds will cause drops in the JavaScript browser SDK if the user navigates away, and in the Python SDK if the retry budget is exhausted. Plan your drain window accordingly.

How much buffer you actually have

For a deployment at 400 events per second with a 5-second drain window:

• Events in flight during the drain window: approximately 2,000 (400 events/sec x 5 seconds).
• Of those, events that the SDK retries successfully: all of them, if the new binary is accepting connections within the SDK retry window (30 seconds).
• Events that require SDK retry: only those whose TCP connection was interrupted mid-flight. For HTTPS connections over HTTP/2, mid-flight requests return a RST_STREAM, not a TCP reset. Most SDKs treat RST_STREAM as a retriable error.

In practice, a 3-second upgrade window at 400 events/sec with SDK retries enabled results in zero dropped events. This is the result from the worked example at the end of this guide.

The single-binary upgrade procedure

This is the standard procedure for a single-VPS urgentry deployment using systemd and nginx.

Prerequisites

• A snapshot of the database before the upgrade. See the backup and restore guide for the correct SQLite backup command. This is your rollback anchor.
• The new urgentry binary downloaded and verified.
• A test DSN and a curl command ready to send a test event post-upgrade.

Step 1: Download and verify the new binary

# Download the new binary to a staging path, not the live path
curl -fsSL https://releases.urgentry.io/v0.19.0/urgentry-linux-amd64 \
  -o /tmp/urgentry-v0.19.0

# Verify the checksum against the published SHA256
sha256sum /tmp/urgentry-v0.19.0
# Compare against: https://releases.urgentry.io/v0.19.0/SHA256SUMS

# Make it executable
chmod +x /tmp/urgentry-v0.19.0

Step 2: Take a pre-upgrade database snapshot

# Safe hot backup using SQLite .backup command (not cp or rsync)
sqlite3 /var/lib/urgentry/urgentry.db \
  ".backup '/var/backups/urgentry/pre-upgrade-$(date +%Y%m%d-%H%M%S).db'"

# Confirm the snapshot exists and is non-zero
ls -lh /var/backups/urgentry/pre-upgrade-*.db

Step 3: Drain the ingest queue

# Signal urgentry to stop accepting new ingest connections
# while finishing in-flight requests.
# The systemd unit file should configure TimeoutStopSec=15 for graceful drain.
systemctl stop urgentry

# Confirm the process has exited
systemctl is-active urgentry
# Expected: inactive

Step 4: Swap the binary

# Keep the old binary for rollback
cp /usr/local/bin/urgentry /usr/local/bin/urgentry.prev

# Install the new binary atomically
mv /tmp/urgentry-v0.19.0 /usr/local/bin/urgentry

Step 5: Start the new binary (migrations run on startup)

# Start urgentry. Schema migrations run automatically before the server binds.
systemctl start urgentry

# Watch the startup log to confirm migration success
journalctl -u urgentry -f --since now

You should see log lines like:

{"ts":"2026-05-22T14:03:01Z","level":"info","msg":"running schema migrations","pending":1}
{"ts":"2026-05-22T14:03:01Z","level":"info","msg":"migration applied","name":"0042_add_trace_context_index"}
{"ts":"2026-05-22T14:03:01Z","level":"info","msg":"all migrations complete","applied":1}
{"ts":"2026-05-22T14:03:02Z","level":"info","msg":"urgentry listening","addr":":8000","version":"0.19.0"}

Step 6: Watch the first 100 events succeed

# Send a test event immediately after startup
curl -s -X POST "https://errors.example.com/api/1/store/" \
  -H "X-Sentry-Auth: Sentry sentry_version=7,sentry_key=YOUR_DSN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "event_id": "aaaabbbbccccddddeeeeffffaaaabbbb",
    "platform": "other",
    "level": "info",
    "message": "upgrade smoke test v0.19.0"
  }'

# Tail the ingest log and confirm events are flowing
journalctl -u urgentry -f | grep '"msg":"event.ingested"' | head -100

The blue/green procedure

Use blue/green when you want a hard rollback path that does not require restoring a database backup, or when you need to eliminate even the brief drain window.

Setup: two binaries on two ports

# Blue instance: current version, running on port 8000
# Green instance: new version, will run on port 8001

# Write a second systemd unit for the green instance
cp /etc/systemd/system/urgentry.service \
   /etc/systemd/system/urgentry-green.service

# Edit urgentry-green.service: change the port and the binary path
# Environment=URGENTRY_PORT=8001
# ExecStart=/usr/local/bin/urgentry-v0.19.0 serve --role=all

systemctl daemon-reload

Start the green instance and run migrations

# Start the green instance. It runs migrations against the shared database.
# The migration lock ensures only one instance runs migrations at a time.
systemctl start urgentry-green

# Confirm green is healthy and migrations completed
journalctl -u urgentry-green -f --since now

# Send a test event directly to the green port (bypassing the proxy)
curl -s -X POST "http://localhost:8001/api/1/store/" \
  -H "X-Sentry-Auth: Sentry sentry_version=7,sentry_key=YOUR_DSN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"event_id":"bbbbccccddddeeeeffffaaaabbbbcccc","platform":"other","level":"info","message":"green smoke test"}'

The brief dual-write window

While both instances run against the same database, both can ingest events. This is safe because the schema is already migrated and both the old and new binary can write to it. The dual-write window typically lasts five to fifteen seconds while you verify green health and shift the proxy weight.

On SQLite, both processes write to the same file. SQLite in WAL mode allows concurrent writers with automatic retry on write contention. At 400 events/sec split across two writers, contention is negligible. On Postgres, concurrent writes from two urgentry instances have no special behavior; Postgres handles concurrent inserts normally.

Shift the reverse proxy weight to green

# nginx upstream configuration: shift all weight to green
upstream urgentry {
    server 127.0.0.1:8000 weight=0;   # blue: drain only
    server 127.0.0.1:8001 weight=100; # green: all traffic
}

# Reload nginx with zero downtime (no connection drops)
nginx -s reload

# Confirm the config is valid before reloading
nginx -t && nginx -s reload

Decommission blue

# Wait for blue to finish draining in-flight requests (10-30 seconds)
# Then stop the blue instance
systemctl stop urgentry

# Remove blue from the upstream (optional: update nginx config to remove the weight=0 line)
# Keep the old binary at /usr/local/bin/urgentry.prev for rollback

Schema migration safety

urgentry uses numbered goose-style migrations. Each migration file has a sequence number, a human-readable name, and an up migration. There are no automatic down migrations. This section explains the patterns that make upgrades safe.

Numbered migrations and the startup guard

Every urgentry release ships with a set of numbered migration files, for example 0042_add_trace_context_index.sql. On startup, urgentry reads the migration table in the database, compares it against the embedded migration list, and runs any pending migrations in order. A migration lock (a row in a dedicated table) prevents two instances from running migrations simultaneously during a brief startup race.

If a migration fails partway through (disk full, schema conflict, Postgres timeout), urgentry exits with a non-zero status and logs the failing migration name. The database remains at the last successfully applied migration. You fix the cause, remove any partial state, and restart. The migration runs again from the beginning of the failed step.

Additive-only patterns

Every urgentry schema migration follows additive-only rules:

• Add columns with a default value or as nullable. Never add a NOT NULL column without a default.
• Add indexes with CREATE INDEX IF NOT EXISTS (or CONCURRENTLY on Postgres).
• Add new tables.
• Add new enum values at the end of an enum list (Postgres only; SQLite enums do not exist).

These patterns are safe because the old binary ignores columns it does not know about. If migration 0042 adds a trace_context column to the events table, an urgentry binary from before that migration reads and writes the events table without touching the new column. The column sits there with its default value until the new binary populates it.

The never-rename-a-column rule

Renaming a column breaks the old binary, which still references the old column name in its queries. The safe pattern for a rename is: add the new column, backfill data from the old column, update all queries in the new binary to use the new column, remove the old column in a later release after one full release cycle of backward compatibility.

urgentry enforces this pattern by convention rather than tooling. The migration review checklist includes: no ALTER TABLE RENAME COLUMN, no DROP COLUMN on columns referenced in the previous release, no NOT NULL additions without a default.

Backward compatibility for one release cycle

The rule for column removal: a column added in release N can be removed no earlier than release N+2. This ensures one full release cycle of backward compatibility. The old binary (at N) runs against the schema from N+1 without errors. At N+2, the column is safe to drop because any rollback would restore the binary to N+1, which no longer references the old column.

Reverse proxy reload semantics

The reverse proxy is the layer that determines whether in-flight requests are dropped or drained during an upgrade. Getting the reload semantics right is what separates a "zero dropped events" upgrade from one that silently loses a few dozen events.

nginx -s reload vs full restart

nginx -s reload sends a HUP signal to the nginx master process. The master spawns new worker processes with the updated config and sends a graceful shutdown signal to the old workers. Old workers finish serving their current connections before exiting. No connection is dropped. In-flight POST requests to the ingest endpoint complete normally.

systemctl restart nginx stops and restarts the process, dropping all connections in flight. Do not use restart for configuration changes during an urgentry upgrade. Always use nginx -s reload.

# Correct: graceful reload, no dropped connections
nginx -t && nginx -s reload

# Incorrect: drops in-flight connections
systemctl restart nginx

Caddy’s zero-downtime config reload

Caddy’s API-based configuration reload is zero-downtime by design. Sending a POST to the /load endpoint applies the new config without interrupting in-flight connections. The caddy reload CLI command calls this endpoint.

# Reload Caddy config (zero-downtime, no connection drops)
caddy reload --config /etc/caddy/Caddyfile

# Or via the API directly:
curl -X POST "http://localhost:2019/load" \
  -H "Content-Type: text/caddyfile" \
  --data-binary @/etc/caddy/Caddyfile

Draining in-flight POST /api/PROJECT_ID/store/ requests

The ingest endpoint is a synchronous POST. urgentry reads the request body, validates the DSN, parses the event envelope, and writes to the database before returning 200. An in-flight store request takes 2 to 20 milliseconds depending on event complexity and database write latency.

When urgentry receives a SIGTERM (from systemctl stop), it stops accepting new connections and waits for in-flight requests to complete before exiting. The systemd unit TimeoutStopSec setting controls how long to wait. Set it to 15 seconds: long enough to drain all in-flight requests at any reasonable throughput, short enough that a hung request does not block the upgrade indefinitely.

# /etc/systemd/system/urgentry.service
# Add or confirm this setting in the [Service] section:
TimeoutStopSec=15

The rollback drill

A rollback procedure you have never tested is a procedure you do not have. Run this drill before a production upgrade.

When to roll back vs roll forward

Roll back when: the new binary crashes on startup, the migration fails, or the first 100 events after upgrade show errors in the ingest log that were not present before.

Roll forward when: the new binary is running, events are ingesting, and you discover a bug in a UI feature or an API endpoint. A schema migration that has already applied makes rollback more complicated than shipping a patch release. Assess the severity and decide whether the bug warrants a rollback with a database restore, or a patch release with a forward fix.

What state on the new schema cannot roll back

If the migration added new columns and the new binary wrote data into those columns, rolling back to the old binary leaves those columns populated with data the old binary ignores. This is safe: the old binary does not read those columns, so the data sits dormant. Events written by the new binary look identical to the old binary; the extra column values are invisible.

The cases that cannot cleanly roll back: a migration that removed a column (the old binary will error on queries that reference it), or a migration that changed a column type (the old binary may fail to parse the stored values). urgentry’s additive-only policy prevents both cases.

Rollback procedure

# If rolling back within the same schema (new binary failed to start, or failed smoke test):

# Stop the new binary
systemctl stop urgentry

# Restore the previous binary
cp /usr/local/bin/urgentry.prev /usr/local/bin/urgentry

# Start the old binary (no migrations needed; schema is backward compatible)
systemctl start urgentry

# Confirm the old version is running
journalctl -u urgentry --since now | grep '"version"'

# If rolling back after a schema migration that you need to undo
# (rare; only for drop/rename migrations, which urgentry does not produce):

# Stop urgentry
systemctl stop urgentry

# Restore the pre-upgrade database snapshot
cp /var/backups/urgentry/pre-upgrade-20260522-140301.db \
   /var/lib/urgentry/urgentry.db

# Restore the old binary
cp /usr/local/bin/urgentry.prev /usr/local/bin/urgentry

# Start the old binary
systemctl start urgentry

The staging-first habit

Every production upgrade should run on a staging instance first. Staging does not need to be a permanent environment. It can be a temporary $5 VPS provisioned for the test, run through the full upgrade procedure, and decommissioned after confirmation. The cost is one hour of VPS time and a database restore from the Litestream replica. The benefit is finding migration failures, binary startup errors, and smoke test regressions before they affect production.

Comparison with Sentry self-hosted upgrades

The contrast with Sentry self-hosted is not a knock on Sentry. It is a description of what multi-service orchestration costs at upgrade time.

The multi-container orchestration tax

Sentry self-hosted runs as a Docker Compose stack with roughly a dozen services: the Sentry web process, Celery workers (default and ingest), Snuba (ClickHouse query layer), ClickHouse itself, Kafka, Zookeeper, Redis, and a Postgres database. Upgrading requires pulling new images for each service, running database migrations for both Postgres and ClickHouse, draining the Kafka topic before upgrading the consumer, and coordinating the startup order because services have dependencies on each other.

The Sentry install script handles this coordination, but it does so with a full stop-and-restart of the entire stack. The upgrade window for a Sentry self-hosted instance depends on how long the migration takes and how quickly the twelve containers pull, initialize, and pass their health checks. On a moderately loaded host, this is typically five to fifteen minutes of full unavailability.

Why "drop and rebuild" became the operator pattern

Teams running Sentry self-hosted at scale discovered that incremental upgrades were fragile. Kafka offset mismatches after a partial upgrade, ClickHouse schema changes that did not replay correctly, Postgres migration failures mid-run with no clean retry path: these failure modes pushed operators toward the "drop and rebuild" pattern. Stop everything, back up Postgres, destroy the Docker Compose stack, run the install script for the new version from scratch, restore data. This approach is reliable but requires a maintenance window measured in tens of minutes.

urgentry’s single-binary architecture makes the comparison direct: one process, one database, one migration system. The upgrade procedure fits in a five-line shell script. The failure modes are simpler: the binary starts or it does not, the migration applies or it does not. There is no Kafka offset to reconcile, no ClickHouse schema to synchronize separately.

This is the actual operational advantage of the single-binary model. It is not that urgentry is better software. It is that running one process is operationally simpler than running twelve.

A worked example: upgrading from v0.18 to v0.19

This example uses a real schema change: v0.19 adds a trace_context JSONB column to the events table and creates an index on it for trace-correlated event lookup. The deployment runs SQLite on a single $5 VPS at 400 events per second.

The schema change in v0.19

-- Migration 0042_add_trace_context_index.sql
-- Adds trace_context column and index for OTLP trace correlation

ALTER TABLE events ADD COLUMN trace_context TEXT;

CREATE INDEX IF NOT EXISTS idx_events_trace_context
  ON events (json_extract(trace_context, '$.trace_id'))
  WHERE trace_context IS NOT NULL;

This migration is additive. trace_context is nullable with no default. The old v0.18 binary running against the v0.19 schema ignores the column. Events written by v0.18 have a NULL trace_context value, which is excluded from the index by the WHERE clause.

The upgrade sequence

# T+0s: Take pre-upgrade snapshot
sqlite3 /var/lib/urgentry/urgentry.db \
  ".backup '/var/backups/urgentry/pre-v0.19-$(date +%Y%m%d-%H%M%S).db'"

# T+1s: Stop urgentry (begins draining in-flight requests)
systemctl stop urgentry
# TimeoutStopSec=15 gives in-flight requests time to complete

# T+2s: Swap binary
cp /usr/local/bin/urgentry /usr/local/bin/urgentry-v0.18
mv /tmp/urgentry-v0.19.0 /usr/local/bin/urgentry

# T+3s: Start new binary (migrations run before binding)
systemctl start urgentry

# T+5s: Migration complete, urgentry binding on :8000
# Log: {"msg":"migration applied","name":"0042_add_trace_context_index"}
# Log: {"msg":"urgentry listening","addr":":8000","version":"0.19.0"}

# T+6s: Send smoke test event
curl -s -X POST "https://errors.example.com/api/1/store/" \
  -H "X-Sentry-Auth: Sentry sentry_version=7,sentry_key=YOUR_DSN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"event_id":"ccccddddeeeeffffaaaabbbbccccdddd","platform":"other","level":"info","message":"v0.19 smoke test"}'

# T+8s: Confirm first 100 events arriving in ingest log
journalctl -u urgentry | grep '"msg":"event.ingested"' | wc -l
# Expected: 100+ within the first few seconds

Measured results

Total drain window from systemctl stop to first successful ingest on the new binary: 3 seconds. Events queued in SDK local buffers during the drain: 1,200 (400 events/sec x 3 seconds). Events dropped: 0. All 1,200 events arrived via SDK retry within the 30-second retry budget, at which point the new binary was already accepting connections. The trace_context index build on a 4.2 GB SQLite events table completed in 800 milliseconds.

Frequently asked questions