Observing Multiple services inside one application

[Bourhjoul]

This is not an issue but more of discussion if that's okay.

We have the following scenario, where we have one service containing multiple workers (a worker is just a rabbitmq consumer of a certain queue), the issue is we want to observe and trace the worker instead of tracing the service, so we can have a better service map and with the current setup of the package this is a limitation because we basically define the service name before runtime.

My questions are:

• is there is a way to acheive this with the package or we should somehow configure our sdk...etc?
• is there is a way to generate a service map based on another attribute instead of service.name?

if I'm asking in the wrong, apologies and would apperciate a little help and guidance.

Thank you!

[trentm]

Hi @Bourhjoul,

same thread

Interesting question. For starters I'm assuming your "workers" are in the same Node.js process and same thread (i.e. not using worker_threads).

With the following example I was able to set separate service.name values reporting telemetry out of the same thread. However the Service Map is not showing a connection between the two. In other words: this was a failure.

const { trace, context, SpanKind } = require('@opentelemetry/api');
const { BasicTracerProvider, BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { resourceFromAttributes } = require('@opentelemetry/resources')
const {OTLPTraceExporter} = require('@opentelemetry/exporter-trace-otlp-proto');

const exporterB = new OTLPTraceExporter();
const providerB = new BasicTracerProvider({
    resource: resourceFromAttributes({
        'service.name': 'serviceB'
    }),
    spanProcessors: [
        new BatchSpanProcessor(exporterB)
    ]
});
const tracerB = providerB.getTracer('b-tracer');
process.once('beforeExit', async () => {
    await providerB.shutdown();
});


async function main() {
    const tracer = trace.getTracer('a-tracer');
    let parentCtx;
    tracer.startActiveSpan('a-span', {kind: SpanKind.SERVER}, (span) => {
        span.setAttribute('foo', 'bar');
        parentCtx = context.active();
        span.end();
    });

    tracerB.startActiveSpan('b-span', {kind: SpanKind.SERVER}, parentCtx, (span) => {
        span.setAttribute('foo', 'bar');
        span.end();
    });
}

main();

Run that via something like:

OTEL_EXPORTER_OTLP_ENDPOINT=...  OTEL_EXPORTER_OTLP_HEADERS="Authorization=..." OTEL_SERVICE_NAME=serviceA node --import @elastic/opentelemetry-node multi-services-issue-978.js

In the example above, I am using the globally-registered (by EDOT Node.js) TracerProvider for "serviceA", and a manually created separate TracerProvider for "serviceB". That way, they can have a separate service.name in their static resource. There is some detail I'm missing to make the Service Map connection. That is possibly tractable.

However, this was with manually created spans (tracer.startActiveSpan) where I could easily control the Tracer that is used (tracerB for spans that should be in "serviceB"). With auto-instrumentation I don't know of a way to make that work in the same thread. We could setup both TracerProviders with the appropriate instrumentation (e.g. instrumentation-amqplib in your case). However, they would both be wrapping the amqplib library (or depending on instrumentation details, one of them might "win"). The result would not be the desired outcome that spans for workerA use the tracer for serviceA, and spans for workerB for serviceB.

separate threads

If your workers are in separate Node.js threads (using worker_threads) then this might be a bit easier. A separate Node.js worker thread has a separate isolated event loop and globalThis allowing separate globally-registered TracerProvider (and hence instrumentations that work independently).

Note that EDOT Node.js currently has a guard on only starting the SDK on the main thread so that would have to be worked-around.