Help debugging GRPC ResourceExhausted error

[KasonBraley]

Looking for some tips on debugging the error message below. As you can see, the size continues to grow on each log statement and doesn't stop. Are there any good ways to get debug output or to get more information onto where these specific messages are coming from?

2023/02/28 20:28:32 max retry time elapsed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4983765 vs. 4194304)

2023/02/28 20:29:33 max retry time elapsed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (6274314 vs. 4194304)

2023/02/28 20:30:33 max retry time elapsed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7896284 vs. 4194304)

2023/02/28 20:31:33 max retry time elapsed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7923905 vs. 4194304)

2023/02/28 20:32:33 max retry time elapsed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7923905 vs. 4194304)

2023/02/28 20:33:33 max retry time elapsed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7923905 vs. 4194304)

[KasonBraley]

I've narrowed it down to the otelhttp package.

{
    "level": "error",
    "ts": 1678140729.5210114,
    "caller": "reporting/otel.go:37",
    "msg": "context deadline exceeded: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (18083865 vs. 4194304)",
    "stacktrace": "main.initOTEL.func1\n\t/go/src/app/services/reporting/otel.go:37\ngo.opentelemetry.io/otel.ErrorHandlerFunc.Handle\n\t/go/pkg/mod/go.opentelemetry.io/[email protected]/error_handler.go:37\ngo.opentelemetry.io/otel.(*delegator).Handle\n\t/go/pkg/mod/go.opentelemetry.io/[email protected]/handler.go:42\ngo.opentelemetry.io/otel.Handle\n\t/go/pkg/mod/go.opentelemetry.io/[email protected]/handler.go:95\ngo.opentelemetry.io/otel/sdk/metric.(*periodicReader).run\n\t/go/pkg/mod/go.opentelemetry.io/otel/sdk/[email protected]/periodic_reader.go:169\ngo.opentelemetry.io/otel/sdk/metric.NewPeriodicReader.func1\n\t/go/pkg/mod/go.opentelemetry.io/otel/sdk/[email protected]/periodic_reader.go:128"
}