feat: minimal tombstone integration #4933

supervacuus · 2025-11-25T09:44:38Z

📜 Description

As discussed last week with @markushi, the process will be to make minimal atomic changes in each PR and merge directly to main rather than accumulating on an uber feature branch. This allows for easier review/feedback/corrections, and we can already test a subset of the entire feature "in the field".

The first minimal change is a basic tombstone integration:

exposes internal option to enable/disable (disabled by default)
the integration will only ever be enabled if the runtime system is at least Android 12
the current setup suggests an operation where users will have to disable NDK support (or only depend on sentry-android-core) or get two reports for the same crash
only retrieves the most current tombstone (the current implementation is not entirely correct: we should either remove any remaining ApplicationExitInfo entries with REASON_CRASH_NATIVE or report them too, including the option for the latter; I left this out for minimal interface exposure in the first step, but either variant is easy to add to this PR or later)
adds a basic tombstone parser/decoder and accompanying snapshot test
introduces the following dependencies:
- runtime: protobuf-javalite (the entire features adds ca. 75KiB to the Android sample release APK)
- build: the protobuf plugin and the protoc compiler to automate protocol updates

Open Issues:

While the protobuf runtime is relatively small, there is still the possibility of conflicting with client-side protobuf versions (major versions often introduce quite severe breakage, but I haven't tested this yet, only reviewed change logs).
~~finding clarity in how to proceed with older tombstones (ignore or handling similarly to ANRv2)~~
~~decision if this minimal setup already makes sense to release as an internal API~~
add ManifestMetadataReader to configure conveniently? (or not since the corresponding options are only internal?)
No changes yet to the UI (there are quite a few aspects that would make these stack-traces more readable, from my pov this is currently out of scope, but i am writing stuff down)

💡 Motivation and Context

First sensible release step for #3295

Part of https://linear.app/getsentry/project/tombstone-support-android-0024cb6e3ffd/

💚 How did you test it?

Added a basic parser snapshot test for a serialized tombstone protobuf.
Manual testing.

📝 Checklist

I added GH Issue ID & Linear ID
I added tests to verify the changes.
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
I updated the docs if needed.
I updated the wizard if needed.
Review from the native team if needed.
No breaking change or entry added to the changelog.
No breaking change for hybrid SDKs or communicated to hybrid SDKs.

🔮 Next steps

Decide if we want to go forward with client-side tombstone processing. This decision should happen with this PR.
- If no, consider sending a tombstone as an attachment, similar to how we attach minidumps and let ingestion/processing deal with decoding (I can prototype this in parallel, since I think we can do most of it in relay, as a first step)
- If yes, decide on adding the protobuf runtime library as a dependency to sentry-android-core (or shade/relocate, or implement our own decoder, given that this is a stable format which only requires a subset of protobuf).
  - Depending on feedback, adaptation to this minimal change (primarily handling older tombstones).
  - Otherwise, integrate an EventProcessor that merges crash events from sentry-native with tombstones.

#skip-changelog (for now)

…ernal)

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

supervacuus · 2025-11-26T21:08:08Z

@markushi, I just realized I cannot omit having a separate "tombstone" marker, even if I report all events, without repeating them. I mean, this was clear to me, but, in addition to it being a must for this PR already, unlike the ANR marker, I must also align it with crashedLastRun.

I wonder if it would make the most sense to do it similarly to ANR by using a TombstoneHint to ensure writing the marker timestamp at the correct life-cycle stage of the event, but introduce a new interface analog to AbnormalExit called CrashExit, so that the hint can take a different route (since it isn't truly an abnormal exit and should affect similar paths to the Native SDK crash marker).

The biggest issue with that approach is that the crashedLastRun is handled entirely in EnvelopeCache rather than AndroidEnvelopeCache, whereas ApplicationExitInfo hint/marker handling happens in the latter.

The PR still makes sense for a first review from you (since, if the general direction makes sense and you have todos not in my list, I can also add a test for the integration itself). Still, I would appreciate a short sync on how to align these execution paths (maybe I don't need to align tombstones with crashedLastRun and can abuse AbnormalExit for the same outcome, though it feels like a significant shortcut, even for minimal internal exposure, which is not at all what I aimed for).

I can convert the PR back to a draft if you prefer.

markushi

Left a few comments - great work so far!

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

sentry-android-core/src/main/proto/io/sentry/android/core/internal/tombstone/tombstone.proto

markushi · 2025-11-27T08:06:57Z

gradle/libs.versions.toml

 otel-semconv = { module = "io.opentelemetry.semconv:opentelemetry-semconv", version.ref = "otelSemanticConventions" }
 otel-semconv-incubating = { module = "io.opentelemetry.semconv:opentelemetry-semconv-incubating", version.ref = "otelSemanticConventionsAlpha" }
 p6spy = { module = "p6spy:p6spy", version = "3.9.1" }
+protobuf-javalite = { module = "com.google.protobuf:protobuf-javalite", version.ref = "protobuf"}


@romtsn Not sure if you followed the conversations, but as you can see here, protobuf requires a runtime dependency. It will have an impact of around 10kb. IMHO fine for now, we should still check how stable this library is to avoid and consumer version mismatch issues.

Co-authored-by: Markus Hintersteiner <m.hintersteiner@gmail.com>

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

sentry/src/main/java/io/sentry/protocol/SentryStackTrace.java

… S, since that is the earliest version where `REASON_CRASH_NATIVE` provides tombstones via the `traceInputStream`)

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

…rker This currently does not work: While we now can optionally enable reporting of "historical" tombstones, by making the `TombstoneHint` `Backfillable` it will automatically be enriched by the `ANRv2EventProcessor` which is currently the only `BackfillingEventProcessor`. The `ANRv2EventProcessor` is partially written in way that is potentially generic for events with `Backfillable` hints, but other parts are enriching as if those are always were ANRs, which up to now was true, but with Tombstones that assumption now breaks. Next Steps: * There is considerable duplication between the ANRv2Integration and TombstoneIntegration

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

…ns via HistoryDispatcher * also update api dumps and formatting for sentry changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

sentry-android-core/src/main/java/io/sentry/android/core/AnrV2EventProcessor.java

…tProcessor this handles all events with Backfillable hint, but adds an interface HintEnricher, to allow hint-specific enrichment (like for ANRs) before and after the generic backfilling happened.

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

markushi · 2025-12-03T20:21:34Z

sentry-android-core/src/main/java/io/sentry/android/core/ApplicationExitInfoEventProcessor.java

+        @NotNull SentryEvent event, @NotNull Backfillable hint, @NotNull Object rawHint);
+  }
+
+  private final class AnrHintEnricher implements HintEnricher {


That sounds like a pretty good solution to me, nice!

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneEventProcessor.java

supervacuus · 2025-12-05T07:55:53Z

...-android-core/src/main/java/io/sentry/android/core/ApplicationExitInfoHistoryDispatcher.java

+  private @Nullable ApplicationExitInfo removeLatest(
+      final @NotNull List<ApplicationExitInfo> exitInfos) {
+    for (ApplicationExitInfo applicationExitInfo : exitInfos) {
+      if (applicationExitInfo.getReason() == policy.getTargetReason()) {
+        exitInfos.remove(applicationExitInfo);
+        return applicationExitInfo;
+      }
+    }
+    return null;
+  }


Not if we exit the loop right after the removal. This is verbatim from the initial code. If we want to remove a potential foot gun RE: future changes, I can make the iterator explicit so that the removal updates both the iterator and the data structure.

… iterator instead of for-each

feat: minimal tombstone integration (disabled by default, options int…

ba0c7db

…ernal)

supervacuus requested review from adinauer, lcian, markushi, romtsn and stefanosiano as code owners November 25, 2025 09:44

sentry bot reviewed Nov 25, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

cursor bot reviewed Nov 25, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

supervacuus commented Nov 26, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java Outdated Show resolved Hide resolved

redo a seemingly missed spotlessApply

9933e47

cursor bot reviewed Nov 26, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

supervacuus added 2 commits November 26, 2025 19:36

use non-plural option + mark integration also as internal + update API

10b1e94

Merge branch 'main' into feat/tombstone_integration

f12b658

cursor bot reviewed Nov 26, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java Outdated Show resolved Hide resolved

markushi reviewed Nov 27, 2025

View reviewed changes

init size exceptions since we know we only need one

4b6e1fb

Co-authored-by: Markus Hintersteiner <m.hintersteiner@gmail.com>

sentry bot reviewed Nov 27, 2025

View reviewed changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java Show resolved Hide resolved

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

supervacuus added 4 commits December 2, 2025 16:20

move instructionAddressAdjustment to SentryStackTrace

95bd726

add copyright notice to tombstone.proto

4f03857

add historical tombstone option

936a2f0

looks like we need to add a TombstoneEventProcessor anyway.

28652e0

cursor bot reviewed Dec 2, 2025

View reviewed changes

sentry/src/main/java/io/sentry/protocol/SentryStackTrace.java Outdated Show resolved Hide resolved

supervacuus added 2 commits December 2, 2025 16:30

...so let's add it to the options if the SDK supports it (adapt to >=…

920221a

… S, since that is the earliest version where `REASON_CRASH_NATIVE` provides tombstones via the `traceInputStream`)

Adapt AndroidEnvelopeCache to also write Tombstone timestamp markers

48ed5e1

cursor bot reviewed Dec 2, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Outdated Show resolved Hide resolved

supervacuus added 4 commits December 2, 2025 16:59

sprinkle with TODOs to highlight next PR steps

6cbfdf8

fix typo

4b1919e

implement TombstoneParser as Closable and close the tombstone stream

d931ddc

supervacuus added 2 commits December 2, 2025 17:48

tighten code with final and null annotations.

65bc76b

eliminate obsolete null check

25f4089

cursor bot reviewed Dec 2, 2025

View reviewed changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java Show resolved Hide resolved

supervacuus added 3 commits December 3, 2025 09:58

Deduplicate ApplicationExitInfo handling for corresponding Integratio…

7db5895

…ns via HistoryDispatcher * also update api dumps and formatting for sentry changes

update tombstone message construction

a8ea230

Merge branch 'main' into feat/tombstone_integration

92cb264

cursor bot reviewed Dec 3, 2025

View reviewed changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java Outdated Show resolved Hide resolved

fix abortMessage check

ed81771

cursor bot reviewed Dec 3, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/AnrV2EventProcessor.java Outdated Show resolved Hide resolved

Merge branch 'main' into feat/tombstone_integration

480bfd6

romtsn mentioned this pull request Dec 3, 2025

Minidump files for native code crashes under Android SDK #4949

Closed

supervacuus added 3 commits December 3, 2025 13:27

Merge branch 'main' into feat/tombstone_integration

c8acdf9

convert AnrV2EventProcessor to a more generic ApplicationExitInfoEven…

125b0c4

…tProcessor this handles all events with Backfillable hint, but adds an interface HintEnricher, to allow hint-specific enrichment (like for ANRs) before and after the generic backfilling happened.

reintroduce, update and correct old inline docs where they make sense.

7e29fbf

cursor bot reviewed Dec 3, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Show resolved Hide resolved

markushi reviewed Dec 3, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneEventProcessor.java Outdated Show resolved Hide resolved

supervacuus added 3 commits December 5, 2025 08:34

remove obsolete TombstoneEventProcessor

605a840

Merge branch 'main' into feat/tombstone_integration

76ef13c

clean up tombstone error handling

7f6dbc9

sentry bot reviewed Dec 5, 2025

View reviewed changes

convert ApplicationExitInfoHistoryDispatcher.removeLatest() to use an…

58a342f

… iterator instead of for-each

Uh oh!

feat: minimal tombstone integration #4933

Are you sure you want to change the base?

feat: minimal tombstone integration #4933

Uh oh!

Conversation

supervacuus commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

supervacuus commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markushi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markushi Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markushi Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

supervacuus Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

supervacuus commented Nov 25, 2025 •

edited

Loading

supervacuus commented Nov 26, 2025 •

edited

Loading

markushi Dec 3, 2025 •

edited

Loading