Hardening a self-hosted error tracker (TLS, auth, audit).

The threat model for a self-hosted error tracker

Before picking controls, name what the attacker wants. An error tracker is a privileged target because of what legitimately flows through it.

What an attacker wants

PII in event payloads. Stack traces often include local variable values. Breadcrumbs capture UI state and network requests. User context fields carry email addresses, user IDs, and in poorly configured SDKs, session tokens. A breach of the event database is a breach of user data from every application you monitor.

Source maps. A source map is a complete map from minified production code back to original TypeScript or JavaScript with full variable names and business logic visible. Attackers who obtain your source maps can read your authentication code, find logic flaws in your API, and understand your data model. Source maps are stored separately from the event database and deserve separate access controls.

The DSN itself. A DSN is a credential. The public key in a DSN authorizes event submission to a specific project. Anyone who holds the public key can send arbitrary events: crafted payloads with malicious data, PII exfiltration via forged error messages, or a simple flood that fills storage and degrades the instance. The private key (used in some Sentry SDK configurations for server-side calls) authorizes more: project reads, release creation, and source map uploads.

Admin UI access. The urgentry admin interface controls DSNs, alert destinations, team memberships, and integration credentials (PagerDuty tokens, Slack webhooks). A compromised admin session is access to all of those.

What an attacker does not care about

The event data itself is not a high-value target for most attacks. Attackers do not need your stack traces to attack your application. They want PII, credentials, and source intelligence. A system that stores events with PII stripped and no credentials in breadcrumbs is a much less valuable target, even if it is publicly reachable.

The ingest endpoint is stateless and write-only for SDK traffic. Compromising it lets an attacker write to your instance, not read from it. The UI and API endpoints are where reads happen. Separating ingest exposure from UI exposure (different firewall rules, different authentication requirements) reduces blast radius.

TLS termination at the reverse proxy

urgentry speaks plain HTTP on port 8000. TLS lives at the proxy. This is the right architectural split: one component manages certificates, the other processes events. The reverse proxy configs guide covers full nginx, Caddy, and Traefik setups. This section focuses on the security-specific directives.

nginx TLS block with TLS 1.3 only

The Mozilla SSL Configuration Generator produces the correct cipher suite list for your target compatibility profile. For urgentry, the modern profile (TLS 1.3 only, no TLS 1.2 fallback) is appropriate unless your SDK runs in an environment that does not support TLS 1.3.

# In your HTTPS server block
ssl_certificate     /etc/letsencrypt/live/errors.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/errors.example.com/privkey.pem;

# TLS 1.3 only (modern profile from Mozilla SSL Config Generator)
ssl_protocols TLSv1.3;
ssl_prefer_server_ciphers off;

# Session cache and tickets
ssl_session_timeout 1d;
ssl_session_cache   shared:SSL:10m;
ssl_session_tickets off;

# OCSP stapling
ssl_stapling        on;
ssl_stapling_verify on;
resolver            1.1.1.1 8.8.8.8 valid=300s;
resolver_timeout    5s;

# HSTS: 6-month max-age to start; add preload and includeSubDomains once confirmed
add_header Strict-Transport-Security "max-age=15768000" always;

# Additional security headers
add_header X-Content-Type-Options nosniff always;
add_header X-Frame-Options DENY always;
add_header Referrer-Policy no-referrer-when-downgrade always;

If you need TLS 1.2 compatibility (older SDKs on mobile, embedded devices), add TLSv1.2 to ssl_protocols and add the intermediate cipher list from the Mozilla generator. Do not include TLS 1.0 or 1.1 under any circumstances.

Caddy: TLS 1.3 by default

Caddy uses TLS 1.3 and the Mozilla modern cipher suite by default. No additional TLS directives are needed. Add the security headers manually:

errors.example.com {
    reverse_proxy localhost:8000

    header {
        Strict-Transport-Security "max-age=15768000"
        X-Content-Type-Options nosniff
        X-Frame-Options DENY
        Referrer-Policy no-referrer-when-downgrade
        -Server
    }

    request_body {
        max_size 20MB
    }
}

Certificate provisioning with Certbot

apt install certbot python3-certbot-nginx
certbot --nginx -d errors.example.com
# Verify the renewal timer is active:
systemctl status certbot.timer

Certificate renewal as a failure mode

Let’s Encrypt certificates expire every 90 days. Certbot installs a systemd timer that renews certificates automatically when they approach expiry (30 days by default). The timer is not infallible. If the domain stops resolving, if the ACME challenge path is blocked by a firewall rule, or if the timer unit fails silently, the certificate expires and urgentry becomes unreachable over HTTPS.

Monitor certificate expiry externally. A free tool like UptimeRobot or a curl cron job that checks the certificate’s notAfter date is enough. Set an alert threshold at 14 days remaining so you have time to investigate before expiry. Let’s Encrypt also rate-limits re-issuance to five failed attempts per domain per week; if you exhaust those retrying a misconfigured renewal, you wait seven days.

# Add to weekly cron: alert if certificate expires within 14 days
0 9 * * 1 root \
  days=$(( ( $(date -d "$(openssl s_client -connect errors.example.com:443 \
    -servername errors.example.com 2>/dev/null \
    | openssl x509 -noout -enddate \
    | cut -d= -f2)" +%s) - $(date +%s) ) / 86400 )) ; \
  [ "$days" -lt 14 ] \
    && echo "TLS cert expires in $days days" \
    | mail -s "urgentry cert expiry warning" ops@example.com

Authentication on the UI

urgentry does not include built-in SSO or SAML today. The UI ships with username/password authentication. For a private deployment used only by a known team, that is workable. For a deployment exposed to the public internet, it is not enough.

Option 1: Tailscale (lowest effort)

Bind urgentry to a Tailscale IP rather than a public interface. Every team member installs Tailscale. The urgentry instance is invisible from the public internet. The admin UI is accessible only to devices on your tailnet.

This is the fastest path for small teams. The tradeoff: urgentry’s ingest endpoint needs to remain public-facing (your SDK clients send events from user devices, not from the tailnet). Keep urgentry bound to both the Tailscale IP for the UI and the public interface for ingest only, and use firewall rules to restrict which paths are accessible from each interface.

# urgentry listening on all interfaces; firewall restricts UI access
# to Tailscale CIDR only (example with ufw)

# Allow ingest from anywhere
ufw allow 443/tcp comment "urgentry ingest (via nginx)"

# Allow SSH from Tailscale CIDR
ufw allow from 100.64.0.0/10 to any port 22 comment "SSH via Tailscale"

# Block direct access to urgentry backend port from public
ufw deny 8000/tcp

# Reload
ufw reload

Option 2: oauth2-proxy in front of the UI

oauth2-proxy sits between your reverse proxy and urgentry. It handles the OAuth 2.0 flow with any standard provider (Google, GitHub, Okta, Azure AD) and passes authenticated sessions through to the upstream. Unauthenticated requests get redirected to the provider’s login page.

oauth2-proxy works well when your team already uses one of the supported providers. The setup involves three moving parts: the provider app registration (client ID, client secret, redirect URI), the oauth2-proxy process or container, and a modified nginx or Caddy config that routes through the proxy.

# oauth2-proxy configuration (environment variables or config file)
# Example: Google provider, restrict to a specific email domain

OAUTH2_PROXY_PROVIDER=google
OAUTH2_PROXY_CLIENT_ID=your-google-client-id
OAUTH2_PROXY_CLIENT_SECRET=your-google-client-secret
OAUTH2_PROXY_COOKIE_SECRET=$(openssl rand -base64 32)
OAUTH2_PROXY_EMAIL_DOMAIN=yourcompany.com
OAUTH2_PROXY_UPSTREAMS=http://127.0.0.1:8000
OAUTH2_PROXY_HTTP_ADDRESS=127.0.0.1:4180
OAUTH2_PROXY_REDIRECT_URL=https://errors.example.com/oauth2/callback

# nginx location block: route through oauth2-proxy
# Replace the main location / block in your HTTPS server block

location /oauth2/ {
    proxy_pass       http://127.0.0.1:4180;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Auth-Request-Redirect $request_uri;
}

location = /oauth2/auth {
    proxy_pass       http://127.0.0.1:4180;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header Content-Length    "";
    proxy_pass_request_body off;
}

location / {
    auth_request /oauth2/auth;
    error_page 401 = /oauth2/sign_in;

    auth_request_set $user  $upstream_http_x_auth_request_user;
    auth_request_set $email $upstream_http_x_auth_request_email;
    proxy_set_header X-User  $user;
    proxy_set_header X-Email $email;

    proxy_pass         http://127.0.0.1:8000;
    proxy_set_header   Host              $host;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
}

Note: apply oauth2-proxy only to the UI paths. The SDK ingest endpoints (/api/<project-id>/store/, /api/<project-id>/envelope/) must remain unauthenticated by oauth2-proxy, because SDK clients do not participate in browser OAuth flows. Split the nginx location blocks to skip auth_request on those paths.

Option 3: HTTP basic auth as a stopgap

Basic auth adds a browser-level username/password prompt in front of the entire domain. It is not a replacement for real authentication management, but it raises the bar above an open admin interface while you set up a proper solution.

# Generate the htpasswd file
apt install apache2-utils
htpasswd -c /etc/nginx/.htpasswd adminuser

# In your nginx server block:
# auth_basic "urgentry";
# auth_basic_user_file /etc/nginx/.htpasswd;

Basic auth over TLS is acceptable. Basic auth over plain HTTP is not: the base64-encoded credentials travel in the clear. Always pair it with HTTPS.

DSN hygiene

A DSN (Data Source Name) is a URL that encodes a project identifier and a key pair. It is the credential your SDK uses to authenticate ingest. Getting DSN hygiene right prevents the most common urgentry security incident: a key commited to a public repository.

Public key vs. secret key

The Sentry DSN format embeds two values: a public key and (in older SDK versions) a secret key. In current Sentry SDK versions (7+), the DSN uses only the public key for ingest. The public key authorizes event submission to a specific project. It does not authorize reads, project configuration changes, release creation, or source map uploads. Those operations require an auth token with explicit scopes, issued separately from the DSN.

The public key is intended to be embedded in client-side code and shipped to end users. A browser-side JavaScript SDK puts the DSN in the bundle. That exposure is acceptable because the key only enables write access to one project’s ingest endpoint. What you must protect is the auth token, not the DSN public key.

Per-project DSNs

Create a separate project in urgentry for each application you monitor. Each project gets its own DSN. This scopes the blast radius: if a DSN leaks, only that project’s ingest is exposed. An attacker who obtains the DSN for your frontend application cannot submit events to your backend service project.

Per-project DSNs also simplify rotation. Rotating a single DSN requires updating one SDK config in one application, not hunting down a shared key across multiple services.

DSN rotation procedure

When a DSN leaks or when you rotate keys on schedule:

1. Open the urgentry UI, navigate to the project, and go to Settings > Client Keys.
2. Generate a new key. Copy the new DSN.
3. Update the SDK config in your application to use the new DSN. Deploy the change.
4. Verify events arrive under the new key in the urgentry issue stream.
5. Disable the old key in urgentry. Wait 24 hours to confirm no SDK traffic still uses it.
6. Delete the old key.

When a key leaks in source history

A key committed to a repository is a key that lives forever in git history, even after you delete the file. Rotation is not optional; it is step one. After rotation:

• Search the repository history with git log -S <old-key> --all to identify affected commits.
• Search CI logs and artifact stores for the old key string.
• If the repository is public, assume the key was indexed by crawlers within hours of the commit. Treat it as fully compromised from the moment of the leak, not from the moment you noticed.
• Check the urgentry ingest log for anomalous event volumes under that project during the exposure window. Unexpected spikes indicate active misuse.

PII scrubbing at ingest

Error events carry user data. Stack traces capture local variables. Breadcrumbs record navigation history and network requests that often contain authentication tokens in URL query strings. Scrubbing PII before it reaches the database is the most effective privacy control, because it prevents storage rather than requiring deletion later.

SDK-side scrubbing with beforeSend

The Sentry SDK fires a beforeSend callback on every event before transmission. This is the right place to strip PII, because the data never leaves the client device. Any scrubbing done here requires no network traffic, no server-side configuration, and no database deletions later.

// JavaScript SDK example
Sentry.init({
  dsn: "https://your-key@errors.example.com/1",
  beforeSend(event) {
    // Strip user email from all events
    if (event.user) {
      delete event.user.email;
      delete event.user.ip_address;
    }

    // Remove request headers that may carry auth tokens
    if (event.request && event.request.headers) {
      delete event.request.headers["Authorization"];
      delete event.request.headers["Cookie"];
      delete event.request.headers["X-Api-Key"];
    }

    // Strip query string parameters with sensitive names
    if (event.request && event.request.query_string) {
      const qs = new URLSearchParams(event.request.query_string);
      ["token", "key", "secret", "password", "api_key"].forEach(p => qs.delete(p));
      event.request.query_string = qs.toString();
    }

    return event;
  },
});

Server-side data scrubbers in urgentry

urgentry applies data scrubbers at ingest. You configure scrubbers in project settings. Scrubbers match by field name (exact match or glob) or by regex pattern across the entire event payload.

Built-in field name patterns that urgentry scrubs by default include common credential fields: password, secret, passwd, authorization, api_key, token, auth. The value in any field with these names is replaced with [Filtered].

Add custom scrubbers for fields specific to your application. If your app passes a field named user_ssn or credit_card, add those field names explicitly. Regex scrubbers handle unstructured values: a credit card number appearing in a string field, an email address in a log message, or a UUID that maps to a user account.

The regex caveats

Regex scrubbers add CPU cost to every ingest operation. A poorly written regex that backtracks on pathological input can stall the ingest pipeline. Two rules: anchor your patterns where possible, and test against real event payloads before deploying. A regex intended to match email addresses that matches too broadly can redact legitimate debug data.

Server-side scrubbers also cannot undo what is already in the database. If an event reached the database before you added a scrubber, the PII is in storage. Urgentry does not retroactively apply new scrubber rules to existing events. For GDPR right-to-erasure compliance, you need a separate process to identify and delete events containing a specific user’s data; scrubbers prevent future exposure but do not address past exposure.

Network exposure

urgentry’s ingest endpoint is designed to be public-facing: SDK clients running in user browsers and on user devices need to reach it. The admin UI is not. Conflating the two in your firewall rules is the most common mistake in self hosted sentry security deployments.

Firewall rules: separate ingest from UI

The proxy handles TLS and routing. The firewall controls which source IPs reach the proxy. A sensible split:

# ufw example: public ingest, private UI
# Ingest paths are /api/* -- open to the world via HTTPS
ufw allow 443/tcp comment "HTTPS: ingest + UI via reverse proxy"
ufw allow 80/tcp  comment "HTTP: ACME challenge redirect only"

# Block direct access to urgentry process port
ufw deny 8000 comment "urgentry HTTP port: proxy-only"

# If you have a known CIDR for SDK traffic (server-side SDKs on your infra):
# You can restrict /api/* to that CIDR at the nginx level with allow/deny directives
# rather than at the firewall, since user-facing SDK traffic comes from arbitrary IPs.

If your SDK traffic comes exclusively from your own servers (no browser-side SDKs, no mobile apps), you can add IP allowlisting at the nginx level on the ingest location. This reduces the attack surface to known infrastructure.

Rate limiting at the proxy

urgentry enforces per-project ingest rate limits, but those limits operate on authenticated requests with valid DSNs. A bot or scanner hitting the ingest endpoint with invalid or no DSN generates load before any urgentry-level limit applies. Add rate limiting at the nginx layer to drop unauthenticated or high-volume traffic at the edge.

# In the http{} block (nginx.conf or a shared conf.d file):
limit_req_zone $binary_remote_addr zone=urgentry_ingest:10m rate=30r/s;

# In your server block, on the ingest location:
location ~ ^/api/[0-9]+/(store|envelope)/ {
    limit_req zone=urgentry_ingest burst=60 nodelay;
    limit_req_status 429;

    proxy_pass http://127.0.0.1:8000;
    proxy_set_header Host            $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    client_max_body_size 20m;
    gunzip off;
}

The bot traffic problem

Public Sentry endpoints attract automated scanning traffic. Bots scan for open Sentry instances, send crafted events to probe for misconfigurations, and sometimes flood ingest endpoints to cause denial of service. urgentry returns the same HTTP response shape as Sentry, so it inherits the same bot interest.

Three controls reduce bot impact: rate limiting (above), requiring a valid DSN signature on all ingest requests (urgentry enforces this by default), and monitoring for ingest spikes from unfamiliar IP ranges. An anomaly on your ingest volume graph is the first signal of active bot activity, often appearing before any urgentry-level alert would fire.

Audit logging

Audit logging answers: who did what, and when? For a self-hosted error tracker, the relevant actions are authentication events, credential changes (DSN rotation, auth token creation), project and team changes, and alert rule modifications.

What urgentry logs today

As of v0.2.11, urgentry produces structured logs for:

• Authentication: successful logins, failed login attempts, session creation and expiry.
• DSN changes: key creation, key disable, key deletion, including the actor and timestamp.
• Ingest errors: malformed envelopes, signature failures, rate limit hits, oversized payloads.
• Migration events: schema migrations on startup.

The application log is structured JSON when you set URGENTRY_LOG_FORMAT=json. Each log line includes ts, level, msg, and relevant context fields. This format is what downstream log shippers expect.

Where the gaps are

urgentry does not yet produce a complete audit trail for every UI action. Gaps as of v0.2.11:

• Member invitation and removal from organizations and teams.
• Alert rule creation, modification, and deletion.
• Integration credential access (reading a Slack webhook URL, a PagerDuty key).
• Source map upload and deletion.
• Data scrubber rule changes.

These gaps are tracked upstream. For teams with compliance requirements today, the workaround is to correlate the application log with database-level audit triggers if you run Postgres, or to run with Postgres and enable pgaudit for full DDL/DML audit coverage.

Sending audit logs to a SIEM via OTLP

urgentry includes native OTLP/HTTP support in the same binary. You can ship logs to any OpenTelemetry-compatible backend (Grafana Loki, Elasticsearch, Splunk, Datadog) by configuring the OTLP exporter. This gives you a forwarding path to a SIEM without a separate log shipper process.

# Environment variables to enable OTLP log export
URGENTRY_LOG_FORMAT=json
URGENTRY_OTLP_ENDPOINT=https://your-otel-collector:4318
URGENTRY_OTLP_HEADERS="Authorization=Bearer your-token"

# If using a local OpenTelemetry Collector, point at it:
URGENTRY_OTLP_ENDPOINT=http://127.0.0.1:4318

# OpenTelemetry Collector config (otelcol-config.yaml)
# Receives from urgentry, ships to your SIEM

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  # Example: Loki
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    default_labels_enabled:
      exporter: false
      job: true
  # Or Elasticsearch:
  # elasticsearch:
  #   endpoints: [https://your-es-cluster:9200]

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [loki]

Secrets in environment variables

urgentry reads all sensitive configuration from environment variables: the database connection string (for Postgres), SMTP credentials for alert email, OTLP export tokens, and S3 credentials for blob storage. Keeping those values out of your repository and out of your process list requires deliberate handling.

What not to do

Do not commit a .env file containing secrets to your repository, even a private one. Do not pass secrets as command-line arguments to the urgentry binary; they appear in ps aux output and in process accounting logs. Do not log environment variables at startup; urgentry does not do this by default, but any custom wrapper script that runs env before starting the binary will expose them.

Docker secrets

In Docker Compose or Swarm, use secrets rather than environment variables for credentials. Docker secrets mount as files inside the container at a predictable path:

# docker-compose.yml
services:
  urgentry:
    image: urgentry/urgentry:latest
    environment:
      # Point urgentry at the secret file path
      URGENTRY_DB_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt
    # Or use external: true for Docker Swarm managed secrets

systemd LoadCredential

For systemd deployments, LoadCredential mounts a file from the host filesystem into the service’s private credentials directory at a path only the service user can read. This keeps secrets out of the unit file and out of the environment.

# /etc/systemd/system/urgentry.service
[Unit]
Description=urgentry error tracker
After=network.target

[Service]
Type=simple
User=urgentry
Group=urgentry
WorkingDirectory=/var/lib/urgentry

# Load secrets from host paths into the service's credential directory
LoadCredential=db_password:/etc/urgentry/secrets/db_password
LoadCredential=smtp_password:/etc/urgentry/secrets/smtp_password

# urgentry reads the credential directory path from this env var
Environment=CREDENTIALS_DIRECTORY=%d

# Or set the env var to the credential file path directly:
# ExecStart reads the file at startup via urgentry's _FILE convention
ExecStart=/usr/local/bin/urgentry serve --role=all

PrivateTmp=yes
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/urgentry

[Install]
WantedBy=multi-user.target

The NoNewPrivileges=yes, ProtectSystem=strict, and ProtectHome=yes directives in the unit file are not optional extras. They prevent urgentry from acquiring new capabilities, writing to system directories, or reading home directories even if the process is compromised. Run urgentry as a dedicated non-root service account.

What not to log

Check any startup scripts or wrapper code for these patterns before deploying:

• env or printenv called at startup (dumps all environment variables including secrets to stdout).
• set -x in bash scripts (traces every command, including those that expand secret variable values).
• Configuration validation code that logs the full config struct (often logs all fields including secrets if the struct has no redaction).

A worked hardening checklist

Run through this list on a new urgentry deployment. Each item takes two to ten minutes. The full list fits in an hour.

1. Provision a TLS certificate with Certbot or Caddy’s built-in ACME client. Confirm HTTPS works before continuing.
2. Set TLS 1.3 only in your nginx ssl_protocols directive, or confirm Caddy is using its default modern profile.
3. Add HSTS with max-age=15768000 (6 months). Extend to 1 year and add preload once the setup is stable.
4. Block port 8000 from public access with a firewall rule. All traffic must flow through the reverse proxy.
5. Restrict the UI with oauth2-proxy, Tailscale, or HTTP basic auth over HTTPS. The admin interface should not be open to the public internet.
6. Create per-project DSNs. One project per application. Do not share DSNs across projects.
7. Add beforeSend scrubbing in every SDK integration. At minimum: strip email, IP address, Authorization headers, and Cookie headers from all events.
8. Configure server-side data scrubbers in each urgentry project for application-specific sensitive field names.
9. Set a rate limit on the ingest location in nginx (30 requests per second per IP, burst of 60).
10. Move secrets to Docker secrets or systemd LoadCredential. Remove any .env file from the working directory and from the repository.
11. Set URGENTRY_LOG_FORMAT=json and configure log shipping to your SIEM or log aggregator. Confirm audit log lines for login events appear in the destination.
12. Set a certificate expiry monitor with an alert threshold of 14 days. Test that the alert fires by temporarily adjusting the threshold to 100 days, confirming the alert, then setting it back.

Frequently asked questions