-
Notifications
You must be signed in to change notification settings - Fork 293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support service name override for DSM checkpoints in Spark context #8077
Draft
kr-igor
wants to merge
4
commits into
master
Choose a base branch
from
kr-igor/dsm-spark-streaming
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 4 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~d566c956e1, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.05 s) : 0, 1049948
Total [baseline] (8.616 s) : 0, 8615530
Agent [candidate] (1.051 s) : 0, 1051125
Total [candidate] (8.592 s) : 0, 8592231
section iast
Agent [baseline] (1.179 s) : 0, 1179326
Total [baseline] (9.19 s) : 0, 9190386
Agent [candidate] (1.174 s) : 0, 1174499
Total [candidate] (9.148 s) : 0, 9147992
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.178 s) : 0, 1177756
Total [baseline] (9.184 s) : 0, 9184216
Agent [candidate] (1.177 s) : 0, 1177470
Total [candidate] (9.179 s) : 0, 9178932
section iast_TELEMETRY_OFF
Agent [baseline] (1.171 s) : 0, 1171023
Total [baseline] (9.173 s) : 0, 9173034
Agent [candidate] (1.176 s) : 0, 1175906
Total [candidate] (9.166 s) : 0, 9165844
gantt
title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~d566c956e1, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (712.077 ms) : 0, 712077
BytebuddyAgent [candidate] (711.739 ms) : 0, 711739
GlobalTracer [baseline] (256.664 ms) : 0, 256664
GlobalTracer [candidate] (255.882 ms) : 0, 255882
AppSec [baseline] (54.925 ms) : 0, 54925
AppSec [candidate] (55.108 ms) : 0, 55108
Remote Config [baseline] (707.493 µs) : 0, 707
Remote Config [candidate] (714.866 µs) : 0, 715
Telemetry [baseline] (10.609 ms) : 0, 10609
Telemetry [candidate] (12.665 ms) : 0, 12665
section iast
BytebuddyAgent [baseline] (828.116 ms) : 0, 828116
BytebuddyAgent [candidate] (825.914 ms) : 0, 825914
GlobalTracer [baseline] (246.657 ms) : 0, 246657
GlobalTracer [candidate] (245.461 ms) : 0, 245461
AppSec [baseline] (58.5 ms) : 0, 58500
AppSec [candidate] (57.827 ms) : 0, 57827
IAST [baseline] (21.686 ms) : 0, 21686
IAST [candidate] (21.254 ms) : 0, 21254
Remote Config [baseline] (663.816 µs) : 0, 664
Remote Config [candidate] (649.585 µs) : 0, 650
Telemetry [baseline] (8.672 ms) : 0, 8672
Telemetry [candidate] (8.467 ms) : 0, 8467
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (827.109 ms) : 0, 827109
BytebuddyAgent [candidate] (827.819 ms) : 0, 827819
GlobalTracer [baseline] (246.784 ms) : 0, 246784
GlobalTracer [candidate] (246.027 ms) : 0, 246027
AppSec [baseline] (58.225 ms) : 0, 58225
AppSec [candidate] (57.933 ms) : 0, 57933
IAST [baseline] (21.473 ms) : 0, 21473
IAST [candidate] (21.399 ms) : 0, 21399
Remote Config [baseline] (649.505 µs) : 0, 650
Remote Config [candidate] (662.616 µs) : 0, 663
Telemetry [baseline] (8.55 ms) : 0, 8550
Telemetry [candidate] (8.576 ms) : 0, 8576
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (822.694 ms) : 0, 822694
BytebuddyAgent [candidate] (826.207 ms) : 0, 826207
GlobalTracer [baseline] (245.706 ms) : 0, 245706
GlobalTracer [candidate] (246.159 ms) : 0, 246159
AppSec [baseline] (57.639 ms) : 0, 57639
AppSec [candidate] (58.092 ms) : 0, 58092
IAST [baseline] (21.036 ms) : 0, 21036
IAST [candidate] (21.249 ms) : 0, 21249
Remote Config [baseline] (645.265 µs) : 0, 645
Remote Config [candidate] (665.094 µs) : 0, 665
Telemetry [baseline] (8.302 ms) : 0, 8302
Telemetry [candidate] (8.488 ms) : 0, 8488
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~d566c956e1, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1053276
Total [baseline] (10.376 s) : 0, 10375780
Agent [candidate] (1.051 s) : 0, 1050815
Total [candidate] (10.477 s) : 0, 10477354
section appsec
Agent [baseline] (1.184 s) : 0, 1184054
Total [baseline] (10.683 s) : 0, 10683277
Agent [candidate] (1.186 s) : 0, 1185749
Total [candidate] (10.685 s) : 0, 10684871
section iast
Agent [baseline] (1.177 s) : 0, 1177333
Total [baseline] (10.934 s) : 0, 10934456
Agent [candidate] (1.179 s) : 0, 1178691
Total [candidate] (10.931 s) : 0, 10931479
section profiling
Agent [baseline] (1.272 s) : 0, 1272142
Total [baseline] (10.783 s) : 0, 10783121
Agent [candidate] (1.272 s) : 0, 1271768
Total [candidate] (10.844 s) : 0, 10844139
gantt
title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~d566c956e1, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (711.013 ms) : 0, 711013
BytebuddyAgent [candidate] (711.677 ms) : 0, 711677
GlobalTracer [baseline] (256.063 ms) : 0, 256063
GlobalTracer [candidate] (255.977 ms) : 0, 255977
AppSec [baseline] (54.866 ms) : 0, 54866
AppSec [candidate] (54.797 ms) : 0, 54797
Remote Config [baseline] (708.619 µs) : 0, 709
Remote Config [candidate] (709.422 µs) : 0, 709
Telemetry [baseline] (15.482 ms) : 0, 15482
Telemetry [candidate] (12.636 ms) : 0, 12636
section appsec
BytebuddyAgent [baseline] (728.113 ms) : 0, 728113
BytebuddyAgent [candidate] (729.065 ms) : 0, 729065
GlobalTracer [baseline] (252.222 ms) : 0, 252222
GlobalTracer [candidate] (252.787 ms) : 0, 252787
AppSec [baseline] (169.736 ms) : 0, 169736
AppSec [candidate] (170.137 ms) : 0, 170137
IAST [baseline] (19.238 ms) : 0, 19238
IAST [candidate] (20.1 ms) : 0, 20100
Remote Config [baseline] (658.676 µs) : 0, 659
Remote Config [candidate] (661.25 µs) : 0, 661
Telemetry [baseline] (8.7 ms) : 0, 8700
Telemetry [candidate] (8.055 ms) : 0, 8055
section iast
BytebuddyAgent [baseline] (827.259 ms) : 0, 827259
BytebuddyAgent [candidate] (829.014 ms) : 0, 829014
GlobalTracer [baseline] (246.45 ms) : 0, 246450
GlobalTracer [candidate] (246.406 ms) : 0, 246406
AppSec [baseline] (58.143 ms) : 0, 58143
AppSec [candidate] (58.015 ms) : 0, 58015
IAST [baseline] (21.3 ms) : 0, 21300
IAST [candidate] (21.106 ms) : 0, 21106
Remote Config [baseline] (666.298 µs) : 0, 666
Remote Config [candidate] (643.513 µs) : 0, 644
Telemetry [baseline] (8.544 ms) : 0, 8544
Telemetry [candidate] (8.459 ms) : 0, 8459
section profiling
BytebuddyAgent [baseline] (700.024 ms) : 0, 700024
BytebuddyAgent [candidate] (700.78 ms) : 0, 700780
GlobalTracer [baseline] (371.461 ms) : 0, 371461
GlobalTracer [candidate] (372.02 ms) : 0, 372020
AppSec [baseline] (54.53 ms) : 0, 54530
AppSec [candidate] (53.687 ms) : 0, 53687
Remote Config [baseline] (675.398 µs) : 0, 675
Remote Config [candidate] (674.866 µs) : 0, 675
Telemetry [baseline] (7.87 ms) : 0, 7870
Telemetry [candidate] (7.837 ms) : 0, 7837
ProfilingAgent [baseline] (95.869 ms) : 0, 95869
ProfilingAgent [candidate] (94.976 ms) : 0, 94976
Profiling [baseline] (95.895 ms) : 0, 95895
Profiling [candidate] (95.001 ms) : 0, 95001
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~d566c956e1, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section baseline
no_agent (1.338 ms) : 1318, 1357
. : milestone, 1338,
appsec (1.755 ms) : 1732, 1779
. : milestone, 1755,
appsec_no_iast (1.763 ms) : 1738, 1787
. : milestone, 1763,
iast (1.505 ms) : 1482, 1528
. : milestone, 1505,
profiling (1.57 ms) : 1545, 1595
. : milestone, 1570,
tracing (1.469 ms) : 1444, 1494
. : milestone, 1469,
section candidate
no_agent (1.372 ms) : 1352, 1392
. : milestone, 1372,
appsec (1.755 ms) : 1731, 1779
. : milestone, 1755,
appsec_no_iast (1.77 ms) : 1747, 1794
. : milestone, 1770,
iast (1.517 ms) : 1494, 1540
. : milestone, 1517,
profiling (1.561 ms) : 1537, 1586
. : milestone, 1561,
tracing (1.489 ms) : 1464, 1514
. : milestone, 1489,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~d566c956e1, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section baseline
no_agent (384.691 µs) : 365, 405
. : milestone, 385,
iast (495.054 µs) : 474, 517
. : milestone, 495,
iast_FULL (657.579 µs) : 636, 679
. : milestone, 658,
iast_GLOBAL (522.498 µs) : 501, 544
. : milestone, 522,
iast_HARDCODED_SECRET_DISABLED (497.547 µs) : 476, 519
. : milestone, 498,
iast_INACTIVE (456.806 µs) : 435, 478
. : milestone, 457,
iast_TELEMETRY_OFF (485.738 µs) : 464, 507
. : milestone, 486,
tracing (455.17 µs) : 434, 476
. : milestone, 455,
section candidate
no_agent (377.93 µs) : 358, 398
. : milestone, 378,
iast (494.963 µs) : 473, 516
. : milestone, 495,
iast_FULL (660.567 µs) : 639, 682
. : milestone, 661,
iast_GLOBAL (526.755 µs) : 505, 549
. : milestone, 527,
iast_HARDCODED_SECRET_DISABLED (501.125 µs) : 480, 523
. : milestone, 501,
iast_INACTIVE (461.956 µs) : 440, 484
. : milestone, 462,
iast_TELEMETRY_OFF (490.803 µs) : 468, 513
. : milestone, 491,
tracing (450.988 µs) : 430, 471
. : milestone, 451,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~d566c956e1, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section baseline
no_agent (14.907 s) : 14907000, 14907000
. : milestone, 14907000,
appsec (14.958 s) : 14958000, 14958000
. : milestone, 14958000,
iast (18.982 s) : 18982000, 18982000
. : milestone, 18982000,
iast_GLOBAL (18.041 s) : 18041000, 18041000
. : milestone, 18041000,
profiling (15.165 s) : 15165000, 15165000
. : milestone, 15165000,
tracing (15.231 s) : 15231000, 15231000
. : milestone, 15231000,
section candidate
no_agent (14.919 s) : 14919000, 14919000
. : milestone, 14919000,
appsec (14.8 s) : 14800000, 14800000
. : milestone, 14800000,
iast (19.152 s) : 19152000, 19152000
. : milestone, 19152000,
iast_GLOBAL (17.846 s) : 17846000, 17846000
. : milestone, 17846000,
profiling (15.629 s) : 15629000, 15629000
. : milestone, 15629000,
tracing (14.777 s) : 14777000, 14777000
. : milestone, 14777000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~d566c956e1, baseline=1.45.0-SNAPSHOT~c99c2c2042
dateFormat X
axisFormat %s
section baseline
no_agent (1.465 ms) : 1454, 1477
. : milestone, 1465,
appsec (2.327 ms) : 2286, 2368
. : milestone, 2327,
iast (2.087 ms) : 2034, 2140
. : milestone, 2087,
iast_GLOBAL (2.123 ms) : 2070, 2176
. : milestone, 2123,
profiling (1.957 ms) : 1915, 1999
. : milestone, 1957,
tracing (1.934 ms) : 1893, 1975
. : milestone, 1934,
section candidate
no_agent (1.467 ms) : 1455, 1478
. : milestone, 1467,
appsec (2.349 ms) : 2307, 2391
. : milestone, 2349,
iast (2.088 ms) : 2035, 2141
. : milestone, 2088,
iast_GLOBAL (2.137 ms) : 2084, 2191
. : milestone, 2137,
profiling (1.977 ms) : 1934, 2021
. : milestone, 1977,
tracing (1.923 ms) : 1882, 1963
. : milestone, 1923,
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
This PR adds a new instrumentation for SparkSession, allowing to override DSM service name. This is needed in cases when Spark is running in cloud environments (such as Databricks) and DD_SERVICE is not set.
Additionally:
Motivation
A lot of Datadog customers use Spark Streaming on Databricks. Without this change all DSM checkpoints are generated using random service names, which is not what customers expect.
Results
Checkpoints set from a Databrcks notebook streaming data from kafka:
[dd.trace 2024-12-10 21:06:43:506 +0000] [dd-data-streams-monitor] DEBUG datadog.trace.agent.core.datastreams.DefaultDataStreamsMonitoring - Flushing [StatsBucket{startTimeNanos=1733864790000000000, bucketDurationNanos=10000000000, backlogs=0}] buckets (databricks.all-purpose-cluster.krigor-spark-streaming-test)