Native crash reporting: minidumps and symbolication.

What a minidump is and why native crashes need one

A SIGSEGV kills the process at the machine instruction level. The operating system stops the program mid-execution, before any language-level cleanup can run. Python’s exception handler never fires. The JVM’s uncaught exception hook never runs. The crash reporting SDK in your application has no opportunity to format and send an error event, because the SDK itself is dead along with the rest of the process.

The kernel, however, has everything. It holds the CPU registers at the moment of the fault, the contents of the stack memory for each thread, and a complete list of every library the process had mapped into its address space. A minidump is a structured extraction of that information: enough to reconstruct the call chain, identify the faulting instruction, and show the state of every thread at the moment of death.

What a minidump does not have is symbol names. A native binary in production has its debug information stripped. The function named processPayment in your source code compiles to a block of machine instructions at address 0x00007f8a3c4d2190. The mapping from that address to the function name, source file, and line number lives in a separate artifact: a dSYM bundle on macOS and iOS, a Program Database (.pdb) file on Windows, a proguard mapping file for Android Java/Kotlin code, or DWARF sections inside a companion .debug file for native ELF binaries.

Minidump symbolication is the process of reading an address from the minidump, finding the debug symbol file that covers the module containing that address, and looking up the corresponding function name and source location. The kernel captures the data; symbolication makes it readable.

The three formats you will deal with

Three crash report formats appear across the platforms where native code runs. Each has a different structure and a different upstream producer, but all three contain the same logical data: thread stacks expressed as memory addresses, plus a module list that maps address ranges to binary identifiers.

Breakpad minidump (.dmp)

The Breakpad minidump format is a Microsoft-originated format that Google’s Breakpad project adopted and extended for cross-platform use. The file structure is a series of tagged streams: a thread list stream, a module list stream, a memory list stream, and an exception stream recording the fault type and CPU state. Each module entry includes a CodeId and a DebugId, the two identifiers used to locate symbol files.

Breakpad minidumps are the format produced by the Crashpad library on macOS, Windows, and Linux. They are also the format symbolicator expects as its primary input. When urgentry receives a file at the minidump endpoint, the file is almost always in this format.

Apple crash report (.ips)

Apple’s crash reports were historically plain text with a fixed layout. Since iOS 15 and macOS 12, the system writes crash reports in a JSON-based format with the .ips extension. The file contains thread backtraces as arrays of frame objects, each with a binary image identifier (a UUID matching the binary’s LC_UUID Mach-O command), an offset within that image, and the symbol name if the system can resolve it at capture time (which it usually cannot for production builds).

Apple crash reports differ from Breakpad minidumps in one important way: they do not contain raw stack memory. They contain a pre-unwound list of frame objects. The unwind work happened in-process using Apple’s runtime unwinder before the report was written. This makes them slightly less complete than a full minidump but sufficient for most crash analysis.

Android tombstone

Android calls its native crash output a tombstone. The tombstone is a text file written by the debuggerd daemon when a native process crashes. It contains CPU register state, a stack backtrace, memory maps, log output from the last few seconds, and a list of open file descriptors. The backtrace is written in a human-readable but machine-parseable format: each line contains an address, a library name, and an offset within that library.

Tombstones are less structured than Breakpad minidumps and require more parser-side interpretation. Android also writes a binary protobuf crash dump (the proto-tombstone format introduced in Android 11) alongside the text tombstone. Symbolicator handles both.

Where each format comes from

Each format originates in a different crash-capture daemon or library. Understanding the producer tells you where to intercept the crash data before it reaches urgentry.

Breakpad’s MinidumpWriter and Crashpad

Breakpad introduced the MinidumpWriter class, which runs in a separate out-of-process handler. When the main process faults, a signal handler forks the handler process and the forked process writes the minidump. This out-of-process design avoids the problem of running code in a process whose heap may be corrupted.

Crashpad is the successor to Breakpad, written by Google to address Breakpad’s limitations. Crashpad has a cleaner handler model (a persistent subprocess called the Crashpad handler that waits for crash events), better cross-platform support, and direct HTTP upload capability. Most new projects use Crashpad. Breakpad remains in use in older codebases and on some embedded platforms where Crashpad’s dependencies do not fit.

Apple’s CrashReporter daemon

On macOS, the ReportCrash daemon catches Mach exceptions from any process and writes the crash report to ~/Library/Logs/DiagnosticReports/ (user processes) or /Library/Logs/DiagnosticReports/ (system processes). On iOS, the equivalent is the system crash reporter daemon; crash reports appear in the device’s diagnostics log and can be retrieved via Xcode Organizer or the Settings app.

The Sentry iOS SDK intercepts crashes before or alongside the system reporter using either KSCrash or a Mach exception handler. The SDK collects the crash data and uploads it to urgentry on the next application launch. The system reporter also writes its report independently; both can coexist.

Android’s debuggerd tombstone writer

On Android, debuggerd is a system service that responds when a native process raises SIGSEGV, SIGABRT, or similar signals. The kernel delivers the signal; the process’s signal handler notifies debuggerd via a Unix socket; debuggerd attaches to the dying process with ptrace, reads the thread state and memory maps, writes the tombstone, and releases the process to die. The tombstone appears in /data/tombstones/ on a rooted device or is accessible via adb bugreport.

The Android NDK crash handler (used by the Sentry Android SDK for native crashes) wraps this process with an in-process signal handler that captures a Breakpad minidump before debuggerd does its work. The minidump goes to urgentry; the tombstone stays on the device.

Capturing minidumps with Crashpad

Crashpad separates the crash handler from the application process. The application starts a subprocess called the Crashpad handler at startup and registers a Mach exception handler (macOS/iOS), a Windows vectored exception handler, or a Linux signal handler that points at the Crashpad handler process. When the application crashes, the handler writes the minidump and uploads it over HTTP.

A minimal Crashpad integration in a C++ binary looks like this:

#include "client/crashpad_client.h"
#include "client/crash_report_database.h"
#include "client/settings.h"

bool InitCrashpad(const std::string& urgentry_dsn) {
  base::FilePath handler_path = GetCrashpadHandlerPath();
  base::FilePath database_path = GetCrashDatabasePath();

  std::map<std::string, std::string> annotations;
  annotations["release"] = APP_VERSION;
  annotations["environment"] = "production";

  // The URL is the urgentry minidump endpoint for your project
  std::string url =
      "https://urgentry.example.com/api/1/minidump/"
      "?sentry_key=YOUR_PUBLIC_KEY";

  std::vector<std::string> arguments;
  // No extra handler arguments needed for basic operation

  CrashpadClient client;
  bool success = client.StartHandler(
      handler_path,
      database_path,
      base::FilePath(),  // metrics path, optional
      url,
      annotations,
      arguments,
      true,   // restartable
      false   // asynchronous_start
  );
  return success;
}

The URL in the StartHandler call is the only thing that changes when moving from Sentry to urgentry. The Crashpad handler writes a Breakpad-format minidump and posts it to that URL with a multipart form upload, the same wire format urgentry expects.

Why not Breakpad? Crashpad has a persistent handler process that survives most forms of process corruption. Breakpad forks a handler at crash time from within the faulting process, which can fail if the allocator or signal delivery mechanism is corrupted. Crashpad’s handler is already running before the crash, so it does not depend on the faulting process’s ability to spawn a child. For new projects, Crashpad is the right choice. Migrate from Breakpad if you encounter crashes that Breakpad misses.

Capturing iOS crash reports

iOS gives you two paths for crash data: the Sentry iOS SDK’s crash handler and MetricKit.

Sentry iOS SDK crash handler

The Sentry iOS SDK uses KSCrash to capture crashes in-process and upload them on next launch. After pointing the SDK at urgentry, no further configuration is required:

@import Sentry;

[SentrySDK startWithConfigureOptions:^(SentryOptions *options) {
    options.dsn = @"https://key@urgentry.example.com/1";
    options.environment = @"production";
    options.releaseName = @"MyApp@2.1.0+412";
    // Crash reporting is enabled by default
}];

The SDK captures the crash on signal delivery, writes a structured event to disk, and uploads it the next time the application launches and can reach the network. The event arrives at urgentry’s standard event endpoint, not the minidump endpoint; the SDK formats the crash as a Sentry envelope with an exception payload.

MetricKit MXCrashDiagnostic

MetricKit delivers crash diagnostics to your application delegate up to 24 hours after a crash. The MXCrashDiagnostic object contains a call stack tree and the exception type but not a full minidump. MetricKit diagnostics supplement the SDK’s crash handler for cases where the crash handler itself could not write to disk (out-of-memory kills, watchdog terminations). To handle them, implement MXMetricManagerSubscriber:

- (void)didReceiveDiagnosticPayloads:(NSArray<MXDiagnosticPayload *> *)payloads {
    for (MXDiagnosticPayload *payload in payloads) {
        for (MXCrashDiagnostic *crash in payload.crashDiagnostics) {
            // Format and forward to urgentry via the Sentry SDK
            SentryEvent *event = [self eventFromCrashDiagnostic:crash];
            [SentrySDK captureEvent:event];
        }
    }
}

Simulator vs device

The iOS simulator runs on macOS and produces macOS-style crash reports with x86_64 or arm64e symbols. The device produces arm64 crash reports with different binary UUIDs. Symbol files built for the simulator will not resolve crashes from a physical device and vice versa. Upload both sets with separate sentry-cli debug-files upload invocations, one for simulator builds and one for device builds. Xcode archives automatically place dSYMs for device builds in the archive; simulator dSYMs are in the build products directory.

Capturing Android tombstones

The Sentry Android SDK captures native crashes through an NDK crash handler that installs signal handlers for SIGSEGV, SIGBUS, SIGABRT, and SIGFPE. When a signal fires, the handler writes a Breakpad minidump and queues it for upload. The upload happens either from the NDK layer or from the Java layer on the next application launch, depending on whether the application process can recover enough to run Java code.

import io.sentry.android.core.SentryAndroid;
import io.sentry.SentryOptions;

SentryAndroid.init(this, options -> {
    options.setDsn("https://key@urgentry.example.com/1");
    options.setRelease("com.example.app@2.3.1+88");
    options.setEnvironment("production");
    // NDK crash handler is enabled by default when sentry-android-ndk is on the classpath
});

The foreground-vs-background race is the main operational concern. A foreground crash kills the process immediately; the upload queued in the NDK layer may not flush before the process dies. The SDK writes the minidump to disk first and uploads on next launch. A background crash (a process that Android kills for memory reasons rather than a signal fault) does not produce a minidump at all; those appear as ANR events, not crashes.

For JNI bridges, the signal handler fires in the native thread. If your crash originates in a native method called from a Java thread via JNI, the minidump will contain the native call stack, but the Java frames above the JNI boundary will be absent. The Sentry Android SDK captures a snapshot of the Java stack separately and attaches it to the event as additional context, so you see both the native frames and the Java call chain that led to the JNI call.

Uploading symbol files to urgentry

urgentry stores symbol files through the same debug-files API endpoint Sentry exposes. The endpoint is covered in urgentry’s compatibility matrix. The sentry-cli debug-files upload command handles the upload for all supported formats.

Set your environment variables to point at urgentry:

export SENTRY_URL=https://urgentry.example.com
export SENTRY_AUTH_TOKEN=your_token
export SENTRY_ORG=your-org-slug
export SENTRY_PROJECT=your-project-slug

Upload dSYMs from an Xcode archive:

sentry-cli debug-files upload \
  --include-sources \
  path/to/MyApp.xcarchive/dSYMs

Upload proguard mapping files for an Android release build:

sentry-cli debug-files upload \
  app/build/outputs/mapping/release/mapping.txt

Upload debug symbols for a Linux native binary (the .so file and its companion .debug file):

sentry-cli debug-files upload \
  --include-sources \
  build/libmyengine.so \
  build/libmyengine.so.debug

Verify the upload with the check subcommand:

sentry-cli debug-files check \
  path/to/MyApp.xcarchive/dSYMs

The check command reads the debug IDs from each file and confirms they are present in urgentry. A missing ID means the upload did not complete or the file was rejected. The most common rejection reason is an unsupported format or a duplicate upload of a file that is already present.

Automate the upload in your CI pipeline immediately after a successful build, before any deployment step. The debug files must be present in urgentry before the first crash from that build arrives; retroactive upload works but leaves a gap during which incoming crashes from that build produce unsymbolicated frames.

The actual symbolication step

urgentry receives the minidump at the /api/PROJECT_ID/minidump/ endpoint, stores it, and forwards it to symbolicator. Symbolicator is a separate binary from the getsentry/symbolicator project. It reads the minidump, extracts the module list and the memory addresses for each thread’s stack, queries urgentry’s symbol store for each module by debug ID, downloads the corresponding symbol file, and resolves each address to a function name, source file, and line number.

The resolution for a single frame works like this:

1. The minidump records that frame N is at address 0x00007f8a3c4d2190 in the module libmyengine.so with debug ID E4A1B2C3-F5D6-7890-ABCD-EF1234567890.
2. Symbolicator queries urgentry: do you have a symbol file with debug ID E4A1B2C3-F5D6-7890-ABCD-EF1234567890?
3. urgentry returns the uploaded .so.debug file for that ID.
4. Symbolicator reads the DWARF sections in the .debug file, looks up the address 0x00007f8a3c4d2190 relative to the module base, and finds the function name, file path, and line number.
5. The resolved frame (processPayment at src/payments/processor.cpp:142) goes back to urgentry as part of the event.

Symbolicator runs this resolution for every frame in every thread in the minidump. The work happens in a separate process, so a slow or large minidump does not block urgentry from accepting new events. Symbolication is asynchronous; urgentry stores the raw minidump immediately and updates the issue with symbolicated frames when symbolicator finishes.

Operator deployment shapes

Three deployment configurations cover most urgentry installations.

Same box: symbolicator alongside urgentry

For small teams or single-server deployments, run symbolicator on the same host as urgentry. Configure urgentry to call symbolicator at http://localhost:3021 (symbolicator’s default port). Symbol files are shared via the local filesystem.

# urgentry configuration (environment variables)
SYMBOLICATOR_URL=http://localhost:3021

# Start symbolicator (runs on port 3021 by default)
symbolicator run --config symbolicator.yml

A minimal symbolicator configuration file:

cache_dir: /var/lib/symbolicator/cache
# urgentry provides symbols via its internal symbol store API
sources:
  - type: sentry
    id: urgentry-local
    url: http://localhost:8080/api/0/
    token: "${SENTRY_INTERNAL_TOKEN}"

On a $5 1 vCPU/1 GB VPS, this configuration works for crash volumes up to a few hundred crashes per hour. Symbolication is CPU-bound (DWARF parsing), so add CPU before RAM when scaling this topology.

Separate worker pool

At higher volumes, run symbolicator on dedicated worker nodes. urgentry sends symbolication requests over HTTP; multiple symbolicator instances can sit behind a simple load balancer. Each instance needs read access to the symbol store, either via a shared filesystem or by fetching symbols from urgentry’s API over the network.

SYMBOLICATOR_URL=http://symbolicator-lb.internal:3021

Symbolicator caches resolved symbol files on disk. Each worker maintains its own cache. A shared NFS or S3-backed cache is not required; warm caches on individual workers are sufficient because symbol files change infrequently and the LRU cache fills quickly on a hot symbol set.

External or hosted symbolicator service

Sentry offers a hosted symbolicator service that urgentry can call. This is useful if you want to offload the DWARF parsing work entirely and do not want to operate a symbolicator deployment. Configure the SYMBOLICATOR_URL to point at the hosted endpoint and provide the appropriate authentication token.

Third-party hosted symbolication services that expose a compatible API also work. The interface urgentry calls is the getsentry/symbolicator HTTP API; any service that implements that API is a valid backend.

The four things that break native symbolication in practice

Symbolication failures in production almost always trace to one of four causes.

Missing debug ID

Every module in a crash must have a debug file uploaded with a matching debug ID. The debug ID is a UUID embedded in the binary at link time (LC_UUID in Mach-O, PDB GUID in PE, build ID in ELF). If the symbol file for a particular debug ID is not in urgentry, that module’s frames show as addresses. The fix is to upload symbol files for every module in the crash, including system libraries and third-party dependencies. For Apple system frameworks, use Apple’s symbol server (symbolicator can fetch from it automatically when configured). For third-party libraries, include their debug symbols in your build output.

dSYM version drift

Each build of your application produces a dSYM with a unique UUID. If you upload a dSYM from build N but crash reports arrive from users still running build N-3, urgentry has no symbol file for the build IDs in those older crashes. Keep symbol files for every build that is still in users’ hands, not just the latest release. A 90-day retention matches most mobile app rollout patterns.

Strip settings that remove all debug info

Xcode’s build settings include several strip options: Strip Debug Symbols During Copy, Strip Linked Product, and Strip Style. The correct setting for production builds is to strip the binary but keep the dSYM. The common mistake is to set Strip Style = All Symbols and also delete the dSYM as a disk-saving step. The result is a shipped binary with no recoverable debug information and no dSYM to upload. Set Debug Information Format = DWARF with dSYM File in Xcode and never delete the dSYM from the archive before uploading.

Apple’s bitcode rebuild

Apple deprecated bitcode for iOS, watchOS, and tvOS in Xcode 14. Before that, App Store Connect would recompile bitcode-enabled binaries with its own optimizer, producing a new binary with new addresses. The dSYM you uploaded at build time would not match the binary Apple distributed. If you shipped bitcode-enabled builds before Xcode 14, you had to download the recompiled dSYMs from App Store Connect (via Xcode Organizer or the App Store Connect API) and re-upload them to urgentry. Bitcode is no longer an issue for new iOS projects using Xcode 14 or later, but if you maintain older apps or debug archive crashes from before the deprecation, you may encounter this mismatch.

A worked end-to-end example

The scenario: a production server binary written in Rust that calls a C++ image processing library via FFI. The server crashes with SIGSEGV during a batch processing job. Crashpad captures the crash and uploads a minidump to urgentry.

The crash setup

The Rust binary embeds Crashpad and starts the handler at startup:

fn init_crashpad(urgentry_url: &str) -> crashpad::CrashpadClient {
    let handler = crashpad::HandlerPath::from_env("CRASHPAD_HANDLER")
        .expect("CRASHPAD_HANDLER must point to crashpad_handler binary");

    let database = std::path::PathBuf::from("/var/lib/myapp/crashes");

    let url = format!(
        "{}/api/1/minidump/?sentry_key={}",
        urgentry_url,
        std::env::var("SENTRY_KEY").unwrap()
    );

    let mut annotations = std::collections::HashMap::new();
    annotations.insert("release".to_string(), env!("CARGO_PKG_VERSION").to_string());
    annotations.insert("environment".to_string(), "production".to_string());

    crashpad::CrashpadClient::start_handler(
        &handler,
        &database,
        &url,
        &annotations,
    ).expect("Crashpad handler failed to start")
}

The C++ image library ships with a .so and a separate .debug file. The CI pipeline uploads both after every release build:

sentry-cli debug-files upload \
  --include-sources \
  build/libimageproc.so \
  build/libimageproc.so.debug

sentry-cli debug-files upload \
  --include-sources \
  build/myserver \
  build/myserver.debug

The crash

A batch job sends a malformed TIFF file to the image library. The library reads past the end of a buffer. SIGSEGV fires in the library’s tile-decoding function. Crashpad captures the crash in under 50 milliseconds, writes a minidump to the database directory, and POSTs it to urgentry.

The raw minidump frame for the fault looks like this before symbolication:

Thread 0 (crashed)
 0  libimageproc.so  0x00007f8a3c4d2190 + 0x12190
 1  libimageproc.so  0x00007f8a3c4a8302 + 0xe8302
 2  myserver         0x0000000000483910 + 0x83910
 3  myserver         0x000000000041ee20 + 0x1ee20

After symbolication

Symbolicator reads the module list, fetches the .debug files from urgentry by their build IDs, resolves each address, and returns the symbolicated frames:

Thread 0 (crashed)
 0  libimageproc.so  tiff_decode_tile + 0x90
                     src/codecs/tiff/decoder.cpp:847
 1  libimageproc.so  tiff_process_strip + 0x302
                     src/codecs/tiff/strip.cpp:213
 2  myserver         process_image_batch:: + 0x910
                     src/batch/processor.rs:142
 3  myserver         tokio::runtime::task::harness::poll_future + 0x20
                     /root/.cargo/registry/src/.../harness.rs:488

The fault is in tiff_decode_tile at line 847 of decoder.cpp. The call came from the Rust batch processor at line 142 of processor.rs. The library has a bounds check missing when the tile size in the file header exceeds the allocated buffer. The TIFF decoder reads the tile dimensions without validating them against the buffer size. The fix is a length check before the read loop.

The full round trip from crash to symbolicated frame in urgentry is under five seconds for a cold symbolicator cache and under one second for a warm cache.

Frequently asked questions