[Node.js] Excessive memory consumption when large data is stored on IncomingRequest object

[lforst]

Description

A user was reporting 2x-3x memory consumption when using the Node.js SDK. I had a chat with them and will summarize the setup and findings here.

• SDK setup is a very ordinary express setup with mostly SDK defaults (traces sample rate: 0.5, 0.1 - doesn't matter, leaks more or less equally)
• Requests to the service are transient
• No long-running requests or anything
• Large objects are stored on the res object (in particular large array of strings containing identifiers)
• The memory snapshot shows that NonRecordingSpans and Spans are holding references to the res object via the _isolationScope field.
• The majority of these references seems to come from NonRecordingSpans.
• Interestingly, and I don't know if this is somehow relevant, the NonRecordingSpans are held by very deep chains of Timeout objects. The user does not define any timeouts manually.

My hypothesis: We are not cleaning up NonRecordingSpans "naturally" and keeping them in the exporter or somewhere and they end up being evicted by a timeout policy. This may be in our code, or this may be in OTELs code.

[mydea]

My best guess (slash hope) is, that this is fixed by #14806, by stopping to put request on the SDK processing metadata.

we link the scope to a span, so while that is around the request will also stay around. Not 100% sure if/where/why the spans themselves keep around, but hopefully that PR will make these things go away generally by not keeping a reference to the request around.

[lforst]

My best guess (slash hope) is, that this is fixed by #14806, by stopping to put request on the SDK processing metadata.

Assuming my hypothesis about us not treating NonRecordingSpans correctly is correct, that would fix a singular symptom but not the underlying problem. We need to investigate why NonRecordingSpans are kept in memory.

[mydea]

Yeah, agree, we should also investigate this, but hopefully it is less impactful.

Some general observations:

• NonRecordingSpans are not passed to span processors at all, so they cannot really come to our span exporter, so this cannot really be the cause here.
• The OTEL context (which holds the otel span) is bound to each sentry scope
• We keep the scope on the span, which is circular but should not matter for memory I guess 🤔
• Other than this I could not find any places in our code where spans are kept around

So overall, I believe if there is a problem we can fix, it must be surrounding keeping of scopes and/core OTEL contexts alive, somewhere, not about the spans themselves 🤔 (though def. not 100% certainty on this statement)