What to test before you turn off Sentry billing.

The cost of getting this wrong

The cancel-and-discover gap is the specific failure mode this guide prevents. It goes like this: the team runs urgentry in parallel for a week, the event counts look close enough, someone cancels Sentry on Friday afternoon, and then on Monday morning an engineer opens a Sentry issue link from a runbook and gets a login wall. Then they check the Slack channel where payment service alerts used to land, and nothing has come through since Friday. Then they find that the CI pipeline that tags releases with sentry-cli has been failing silently since the --url flag was never updated.

None of these gaps are hard to close. All of them are trivial to find before the cutover. After the cutover, each one costs time during an incident.

What you cannot recover once Sentry billing stops: Sentry’s historical event data becomes inaccessible on a timeline set by your plan tier, not by you. Webhook integrations that other tools registered against sentry.io stop receiving payloads. Issue permalinks that live in runbooks, Notion pages, and Jira tickets become dead links. None of these can be retroactively fixed from urgentry’s side; they require you to have acted before the billing stopped.

The four classes of test

Functional parity tests confirm that urgentry receives, processes, and groups the same errors your production services generate. They cover the event path from SDK to UI: stack trace readability, breadcrumb population, fingerprinting consistency, and context fields. A passing functional parity test means the data your on-call engineers need during an incident is actually there.

Alert delivery tests confirm that the notifications your team relies on to know something is wrong actually fire. An error tracker that receives events but does not page anyone is a passive log, not an incident response tool. Alert delivery tests are the class most likely to reveal gaps, because alert rules are the part of Sentry that accumulates the most undocumented configuration over time.

Retention coverage tests confirm that you have a plan for the historical event data that urgentry will never have. urgentry starts fresh from the first event it receives as primary backend. Historical Sentry data does not migrate automatically. The retention coverage test is not a technical test; it is a decision: what do you need, where does it go, and who owns it?

Billing closeout covers the administrative steps that convert a running Sentry account into a canceled one without losing access to things you need. It includes the export window, the cancellation timing relative to your billing cycle, and the audit trail you should keep for your own records.

Functional parity tests

Run these tests against urgentry configured as your primary ingest target, not alongside Sentry. You want to confirm that urgentry alone carries the full load, not that urgentry works when Sentry is also available.

Throw an error from every production service

For each service in your production stack that has a Sentry DSN, trigger a real exception path and confirm the event appears in urgentry within a few seconds. Use a real exception class, not a captureMessage call. The goal is to confirm that the ingest path for that specific SDK version, transport configuration, and network path works end to end.

Do not batch this into one representative service. Services often differ in SDK version, initialization options, or proxy configuration. A payment service might use a different SDK release than an internal worker; each one needs a separate confirmation. Mark each service in your tracking doc as confirmed or open.

Throw a handled error

Unhandled exceptions are easy to test. Handled errors, captured with captureException inside a try-catch block, are where SDK configuration differences surface. Trigger at least one handled error per service type and confirm it lands in urgentry with the same context fields you expect: the error class, the message, the stack, and any tags or user context your SDK populates.

Trigger a frontend exception

In a browser or mobile context, trigger a real JavaScript or native exception. Confirm the event reaches urgentry with a readable stack. If your frontend uses minified JavaScript, confirm that source maps are uploaded to urgentry and that the stack trace shows original file paths and line numbers. A minified stack is not a passing test; it means source map upload is broken.

Trigger a backend exception with custom context

Send an exception from a backend service that attaches custom context: a user ID, a tenant identifier, a request ID, or any tag your on-call engineers use to filter issues. Open the event in urgentry and verify every custom field appears. Missing context fields mean your SDK’s scope configuration did not transfer cleanly to the urgentry DSN setup.

Verify breadcrumbs, fingerprinting, and releases

Open an event that should have breadcrumbs, such as an HTTP request that logged several steps before failing. Confirm the breadcrumb trail is present. Check that the issue it landed in has the fingerprint you expect; compare it to the same issue in Sentry if you ran a parallel evaluation. Confirm the release tag on the event matches the release string your CI pipeline tags. Incorrect release strings break source map lookup, which breaks stack traces, which breaks triage.

Alert delivery tests

Before running any alert delivery test, export your full Sentry alert rule list from Settings and document it. Most teams discover rules in this step that nobody on the current team created and nobody knows what they are for. Triage the list into on-call-critical (must fire, must page), team-informational (Slack notification, nice to have), and legacy (can be removed).

Verify Slack, PagerDuty, and email rules fire on urgentry alerts

For each on-call-critical alert rule, trigger the exact condition the rule watches for in urgentry. Do not use the urgentry “send test notification” button alone; it confirms the webhook URL is reachable but not that the trigger condition works. Trigger a real event that satisfies the rule, then watch the destination channel or PagerDuty incident log. Confirm the notification arrives with the right content.

The most common failure: urgentry’s default “new issue” trigger fires on the first occurrence of a fingerprint and then stays silent for subsequent occurrences. If your Sentry rule used an occurrence-count threshold (more than 10 events in 5 minutes), you need to recreate that threshold explicitly in urgentry. A rule that was implicitly count-based in Sentry will not behave the same way in urgentry unless you configure it to.

Update the on-call rotation

If your PagerDuty or OpsGenie service is configured to receive alerts from a Sentry webhook URL, update it to point at urgentry before the cutover. Test it with a real alert. Do not leave the Sentry webhook as a backup; once Sentry billing stops, that webhook stops sending. The rotation needs to work from urgentry alone before the cutover date.

Check the alert noise budget

Run urgentry’s alert rules for at least three days against real traffic and review the alert volume. A rule that was tuned for Sentry’s grouping behavior may produce more noise in urgentry if fingerprinting differs. The goal is not zero alerts, but a volume your on-call rotation can act on without alert fatigue. Adjust thresholds based on what you see, not what you expect.

Retention coverage tests

urgentry starts fresh. The first event it receives as primary backend is event one in its database. Everything that happened before in Sentry exists only in Sentry. The retention coverage test answers: what do you actually need from that history, and do you have it before the access window closes?

Decide what historical data you need to archive

Most teams need less than they think. The data with real operational value is: the fingerprints for recurring issues, so you can recognize a regression when it resurfaces; the timeline of issue first-seen dates, for compliance or post-mortem reference; and the text of issues linked from runbooks or Jira tickets, so those links resolve to something. Raw event payloads from six months ago are rarely worth the storage cost.

Sentry provides a data export via Settings > Data Export. Export the issue list (with fingerprints, titles, first-seen, and last-seen dates) before you cancel. This export is a CSV that fits in a spreadsheet. It is not a full event payload archive, but for most teams it is everything they will ever look up from history.

Understand urgentry’s retention configuration

urgentry’s event retention window is set per project in Settings. The default differs from what a Sentry Team or Business plan provides. Set the retention window to match your compliance or security team’s requirement before the cutover, not after the first audit. Confirm the setting is in writing and reviewed by whoever owns compliance for your organization.

urgentry uses your disk. Unlike hosted Sentry, the retention boundary is enforced by the database cleanup job, which runs on your schedule against your storage. Run the capacity projection before the cutover: event volume per day times average payload size times retention days, plus source map and debug file storage. For a mid-size web service with 90-day retention and source maps, plan for 20 to 100 GB. Run out of disk and ingest stops.

Document where the rolloff data goes

If your compliance framework requires retaining error event metadata beyond urgentry’s retention window, write down where it goes. Options include: the Sentry CSV export archived to cold storage, a periodic urgentry event export to an S3 bucket via urgentry’s API, or a decision that the data does not need to be retained. All three are legitimate. The one that is not legitimate is leaving this undecided until someone asks for it during an audit.

Billing closeout

The billing closeout is not a technical test. It is a sequence of administrative steps that, if done in the wrong order, cost you access to data or money.

Timing the Sentry SaaS cancellation

Cancel at the end of a billing cycle, not mid-cycle. Most Sentry plans do not prorate refunds for the remaining days of a paid period. If your cycle renews on the 1st, cancel before the renewal date, not after. Confirm in your Sentry billing settings when the next renewal date is and plan the cutover date accordingly.

Do not cancel the Sentry account on the same day as the cutover. Wait at least three days after the cutover is confirmed stable, then cancel. That buffer gives you time to discover gaps while Sentry data is still accessible. A three-day buffer is the minimum; a week is better if your billing cycle allows it.

Use the data export window Sentry provides

Sentry provides a data export window after subscription cancellation. The exact duration depends on your plan tier and current Sentry policy; check the Sentry docs for your account type before canceling. Do not assume the window is long. Run the issue export before you cancel, not after the billing stops, so you are not racing against a countdown.

The data export runs from Settings > Account > Data Export in Sentry’s UI. For large organizations with many projects, each project exports separately. Start the exports early; large exports can take hours to prepare.

Keep an audit trail

Write down: the date Sentry billing was canceled, the date each project’s data export was completed, and where the export files are stored. This document is the record that answers “when did we stop using Sentry and where did the data go” when someone asks in six months. Store it where your postmortem docs live, not in a personal drive.

The 10-day checklist

This day-by-day plan assumes a cutover date on day 10. Adjust dates to fit your billing cycle and team schedule.

• Day 1 (cutover minus 10). Confirm urgentry is running as primary backend for at least one non-critical service. Export the full Sentry alert rule list. Triage rules into on-call-critical, informational, and legacy.
• Day 2. Run the functional parity test for every production service. Confirm events land with readable stacks, correct environment tags, and populated context fields. Log any service that fails as an open item.
• Day 3. Recreate every on-call-critical alert rule in urgentry if not already done. Trigger each rule condition manually and confirm the notification reaches its destination.
• Day 4. Run the frontend exception test. Confirm source maps produce readable stacks in urgentry. Confirm mobile symbolication (dSYM or ProGuard) works if applicable.
• Day 5. Review urgentry alert volume from days 2 to 4. Tune thresholds where urgentry is producing more noise than Sentry did for the same traffic pattern.
• Day 6. Start the Sentry data export for all projects. Confirm the export files arrive and are stored in an accessible location. This can take time for large organizations; do not start it on day 9.
• Day 7. Update all CI/CD pipeline steps that reference sentry.io URLs or use sentry-cli without an updated --url flag. Confirm release tagging reaches urgentry after a deploy.
• Day 8. Audit runbooks, Notion pages, and Jira tickets for Sentry issue links. Replace or annotate links that will break after cancellation. This step takes longer than you expect; start it today, not day 9.
• Day 9 (cutover minus 1). Walk the primary triage workflow in urgentry with one on-call engineer. Assign a real issue, read the stack, mark it resolved. Confirm the three sign-off criteria below are all met. If any are not, delay the cutover one billing cycle.
• Day 10 (cutover). Switch all remaining services to the urgentry DSN. Verify the first event from each service. Monitor urgentry’s ingest error log for the first hour. Do not cancel Sentry today.
• Day 11. Normal operations. Watch urgentry for any alert gaps or event processing anomalies.
• Day 12. Spot-check five specific events in urgentry by triggering known error paths. Confirm stacks, context, and breadcrumbs look correct across all services.
• Day 13 (cutover plus 3). If everything is stable, disable Sentry projects (do not delete; historical data stays accessible). Cancel billing if the billing cycle is ending. Confirm the cancellation confirmation email arrives. File the audit trail document.

Three sign-off criteria you cannot skip

The cutover is not ready until all three of these are true. Each one is binary: confirmed in writing, or not confirmed.

1. Error counts match within 2% noise

Pull the event count from urgentry and Sentry for the same 24-hour window during the test period. A difference under 2% is acceptable noise from SDK sampling jitter, retry deduplication, and clock skew. A consistent gap above 2% means something is dropping events on one side. Find it and fix it before the cutover. Do not rationalize a 6% gap as “close enough”; that gap will grow when Sentry is no longer present as a comparison point.

2. Every critical alert fired at least once in urgentry

Review the urgentry alert history for the test period. Every on-call-critical alert rule must have fired at least one real notification to its destination during the test window, not just a test ping. “The rule exists in urgentry” is not a passing criterion. A rule that exists but has never fired may have a misconfigured trigger condition, a webhook URL that returns 200 without actually delivering the notification, or a threshold that has not been met by test traffic. Fire each one against real or deliberately-triggered traffic before signing off.

3. Retention plan is written down

The retention plan must exist as a written document before the cutover, not as a verbal agreement. It should state: urgentry’s retention window setting for each project, the location of the Sentry data export, and who owns the backup strategy for urgentry’s SQLite or Postgres database. A missing retention plan is not a technical gap; it is a compliance and operational risk that materializes the first time someone needs historical data that no longer exists.

The post-cutover regret list

Teams that have completed the migration consistently report the same surprises in the first week after canceling Sentry. These are not failure modes; urgentry is working. They are gaps in the pre-cutover test plan that this guide is designed to close.

Legacy integrations nobody knew about. A service that was instrumented three years ago by an engineer who has since left the company still has a Sentry DSN. It stops sending to anything after the cutover because the DSN points at a canceled Sentry project. The fix is simple, but finding the service takes time. Audit every service with a SENTRY_DSN environment variable before the cutover, not after.

Slack hooks that pointed at sentry.io. Several teams discover that a Slack channel had a Sentry integration configured at the Slack level (not via Sentry’s alert rules) that was routing issue notifications directly. These integrations stop working when the Sentry project is canceled. They do not appear in urgentry’s alert rule list because they were never in urgentry’s alert rule list. Check Slack’s app settings for any Sentry app installations in channels your team uses for monitoring.

Historical issue links in runbooks. A runbook that says “see this Sentry issue for context on the payment service timeout bug” with a sentry.io URL is now a dead link. Runbooks are the documents people read during incidents. Dead links during incidents are expensive. Day 8 of the 10-day checklist exists specifically for this reason.

Release deploy notifications in CI. The sentry-cli releases deploys new command in a CI pipeline that was not updated still points at sentry.io. It starts failing silently. No release tags reach urgentry. Source map lookups start failing because urgentry cannot match release versions to uploaded artifacts. Check every CI pipeline step that mentions sentry-cli and update the --url flag.

The on-call engineer who never used urgentry before an incident. The parallel run happened, the sign-off criteria were met, but two engineers on the on-call rotation never actually opened urgentry during the test period. Their first exposure to the urgentry UI is during a production incident. Walk every on-call engineer through a real triage session in urgentry before the cutover, not just the lead who ran the migration.

FAQ

Sources

• Sentry data export and retention documentation — the data export window Sentry provides after cancellation and the controls for event retention by plan tier.
• FSL-1.1-Apache-2.0 license text — the license under which urgentry is distributed.
• urgentry compatibility audit — the source-scanned audit of 218 documented Sentry API operations and urgentry’s coverage of each one.
• The Sentry switch proof — the thirty-minute protocol for confirming SDK-to-urgentry ingest works before running the full pre-cutover test plan.
• Sentry compatibility checklist — the systematic category-by-category pre-cutover audit covering ingest, event processing, alerts, retention, and integrations.