Consider sensitivity of `error|exception.message` and span status description

[lmolkova]

See open-telemetry/opentelemetry-specification#3496 for the context.

TL;DR: exception messages may contain emails, user info, passwords / secrets and other sensitive information.

Our current (in development) error guidance recommends recording exception / error messages on spans status description or exception | error.message attribute on other signals.

What can we do:

• Option 1: status quo.
  • Keep recording them by default
  • It's user responsibility to sanitize them all on spans and events
  • We'll document that all of them (span status description, exception.message and error.message) may contain sensitive information
• Option 2: don't populate error | exception messages by default unless non-sensitivity is somehow guaranteed. Allow users to opt-in.
  • Note: global opt-in is probably not helpful. The exception | error messages are very useful for observability and it's rather a bug and an edge case when they contain sensitive details.
• Option 3: limit where sensitive info appears to error.message attribute
  • Document it could have sensitive details
  • Don't recommend to populate span status description in semconv ever, eventually deprecate exception.message
  • Users would need to manually (via custom processor) sanitize known occurrences of problematic error messages using Span/Log processor
• ?

[lmolkova]

I think it's the question of picking the least bad solution.

My preference is on Option 3:

• error | exception message is super-useful for observability. OTel already recommends recording it in exception.message (stable)
• it's a bug when error message contains sensitive info
• Having 3 places to put error message info is not great, we can try converging it to a single place with Update recording errors doc - document when to use error, exception (or both) attributes #2296 and OTEP: Recording exceptions as log records opentelemetry-specification#4333. Span-event deprecation gives us opportunity to put exception.message to rest. This would somewhat simplify sanitization and would have positive effect on correlation things across signals