The self-hosted Sentry upgrade treadmill: hard stops and broken migrations

The short version of the problem

Most software upgrades by pulling a newer build and restarting. Self-hosted Sentry does not work that way. The installer reads your current version, checks it against a list of mandatory checkpoints, and stops if you are trying to jump too far ahead. To reach a recent release from an old one, you walk the upgrade through every checkpoint in between, in order, each with its own shutdown and migration run.

Sentry calls these checkpoints hard stops. The official upgrade docs are blunt about it: "When upgrading one must upgrade to each hard stop to pick up significant database changes." That sentence is the whole problem in miniature. The migrations are not safe to replay across arbitrary version gaps, so the project encodes the safe path as a fixed sequence and enforces it at install time.

What a hard stop actually is

A hard stop is a release that introduced a database change significant enough that later migrations assume it already ran. Picture a migration written for version N+5 that adds a column to a table version N+2 created. Jump straight from N to N+5 and the table is not there yet, so the migration fails. The hard stop at N+2 guarantees the table exists before anything downstream touches it.

So the rule is not "upgrade often." The rule is "never step over a hard stop." You can move between two adjacent hard stops in one go, but you have to physically land on each hard-stop release, run its migrations to completion, confirm the stack is healthy, and only then move to the next.

The chain you cannot skip

As of early 2026 the documented hard stops are:

9.1.2
21.5.0
21.6.3
23.6.2
23.11.0
24.8.0
25.5.1

If your instance is on 23.6.2 and the current release is past 25.5.1, you do not have one upgrade ahead of you. You have at least three: 23.6.2 to 23.11.0 to 24.8.0 to 25.5.1, then on to current. Each arrow is a checkout, an installer run, a full stack shutdown, a migration pass, and a health check before the next one. The docs state the cost plainly: "we shut down all the services and then run some data migrations, so expect to have some downtime." There is an experimental --minimize-downtime flag, but the default expectation is that the instance is offline for the length of each migration.

When the chain breaks: three failure modes

The mechanism works when every checkpoint is reachable and every migration completes. Three things break that assumption, and all three show up repeatedly in the getsentry/self-hosted issue tracker.

1. "It looks like you've skipped a hard stop"

The installer guards the chain with an explicit check. Trip it and you get:

Error: It looks like you've skipped a hard stop in our upgrade
process. Please follow the upgrade process here:
https://develop.sentry.dev/self-hosted/releases/#hard-stops

The maddening version of this is when you did follow the path. Issue #3816 documents an upgrade from 24.8.0 to 25.5.1, two adjacent hard stops, that still aborted at the migration step with dcr web upgrade exiting non-zero. Following the documented sequence is necessary, and it is not always sufficient.

2. The hard-stop image that isn't there

You cannot land on a hard stop whose container image has been pulled from the registry. Issue #4015 reports exactly this: an operator on 24.8.0 trying to reach 25.5.1 found the 25.5.1 images missing from both Docker Hub and GHCR, with the oldest available tag being 25.6.2, which is not a hard stop. The required stepping stone had been removed, leaving no compliant path forward. Old instances are the most exposed here, because the hard stops they need are the oldest, and the oldest images are the first to be cleaned up.

3. The migration that dies halfway

Each hop runs migrations against Postgres, ClickHouse (through Snuba), and Kafka topic creation. Any one of them can fail partway. Issue #2582 describes an instance that went from 23.5.0 to 23.11.0 and skipped the 23.6.2 hard stop sitting between them, leaving the operator asking how to remediate after the fact. Once a migration has half-applied, neither the old version nor the new one sees the schema it expects, and recovery is manual database work or a restore from backup.

The pattern this produces: rot, then rebuild

Put the three failure modes together with the fact that most teams upgrade their error tracker rarely, and a predictable pattern emerges. The instance works, so nobody touches it. A year passes. Two years pass. By the time someone needs a feature from a current release, the gap is four or five hard stops wide, several of the required images are gone, and the safest move is no longer to upgrade at all.

This is not hypothetical. In May 2026 pixiv's engineering team published a post titled, roughly translated, "We rebuilt our too-old self-hosted Sentry," describing an instance that had aged past the point of incremental upgrade and had to be stood up fresh. On the other side of the experience, a widely shared writeup titled "I upgraded our 2.5-year-old self-hosted Sentry without losing a single byte" reads less like a maintenance note and more like a war story, which tells you how the community regards the job. One operator summed up the rhythm of it on X: "self-hosted sentry releases break migrations often enough that the maintenance is bursty not steady." The work is not constant. It just lands all at once, when you can least afford it.

Why the architecture forces this

The treadmill is a property of the design, not a bug someone forgot to fix. Self-hosted Sentry is a distributed system: Postgres for relational data, ClickHouse for the event store, Kafka for the ingest pipeline, Snuba as the query layer over ClickHouse, plus Relay, background workers, crons, and the web tier. That is roughly twenty containers coordinated by Docker Compose. A single schema change often has to land in Postgres, ClickHouse, and the Kafka topic layout together, in a specific order. The more moving parts a migration touches, the harder it is to make safe across arbitrary version gaps, and the more the project has to lean on a fixed checkpoint chain to keep the combinations tractable.

There is nothing wrong with that engineering. It is the honest cost of running a system built to scale to enormous event volumes. But most self-hosters are nowhere near that volume, and they inherit the upgrade complexity of a system sized for a problem they do not have.

What a single binary changes

The upgrade story looks different when the tracker is one process against one database. urgentry ships as a single Go binary with SQLite by default and Postgres optional. There is no ClickHouse to migrate in lockstep, no Kafka topic layout to recreate, no Snuba layer to keep in version sync. An upgrade is: stop the old binary, start the new one. Migrations run on startup.

The deeper difference is the migration discipline. urgentry's schema migrations are additive-only and numbered. A new binary runs fine against a schema that already has columns it does not recognize, and an old binary runs fine against a schema with columns it has never heard of. That property removes the reason hard stops exist: there is no migration that assumes an intermediate release, because every migration only adds, never rewrites. You can move several versions in one step, and rollback is putting the old binary back, with no database restore.

# Self-hosted Sentry, three hard stops behind
./install.sh   # land on 23.11.0, migrate, verify, restart
./install.sh   # land on 24.8.0,  migrate, verify, restart
./install.sh   # land on 25.5.1,  migrate, verify, restart
# only now can you upgrade toward current

# urgentry, any number of versions behind
systemctl stop urgentry
cp urgentry-new /usr/local/bin/urgentry
systemctl start urgentry   # migrations run on boot, one step

This is the same trade the rest of urgentry makes. It does not try to be a platform for every observability signal at planetary scale. It runs the Sentry SDK ingest path and the REST API on hardware you can fit on a $5 VPS, and the upgrade model falls out of that simplicity.

If you are staying on Sentry self-hosted

Most teams reading this are not going to migrate today, and there are sane ways to keep the treadmill from running you over.

• Upgrade on a schedule, not on demand. Hopping one hard stop every few months is routine. Hopping five at once is the war story. Put it on the calendar before the gap gets wide.
• Snapshot before every hop. A full backup of Postgres and the ClickHouse volume before each install.sh run turns a half-applied migration from a disaster into a short restore.
• Pin the images you will need. If your path crosses 24.8.0 and 25.5.1, pull and store those images now, before they age out of the registry. Issue #4015 is what happens to operators who assume the images will always be there.
• Read the release notes for the migration warnings, not the features. The breaking changes live in the migration section. SKIP_SENTRY_MIGRATIONS is a documented escape hatch for a wedged upgrade, but using it means you own the schema consistency it was protecting.

The honest framing: the upgrade treadmill is a recurring tax on running a system bigger than your traffic needs. Whether that tax is worth paying comes down to whether you are actually using the scale it buys.

Frequently asked questions