Scope note: measurements in this document describe the generic base variant under
third_party/rgo/v0_60_0/base. Public consumer patch profiles are verified separately bytools/dev/verify_rules_go_profiles.py.
This note captures the first real timing pass for the vendored
third_party/rgo/v0_60_0/base fork using the probe instrumentation added
to the builder and Orchestrion extension paths.
Unlike rules_go_orchestrion_performance_analysis.md, this document is based on measured local runs, not just code reading.
Measurement date: 2026-03-26
Latest baseline refresh: 2026-03-27
These measurements came from a local consumer-style run in the sibling fixture repository:
- workspace:
../rules_test_optimization_tests - target:
//src/go-project:hello_test - host Go:
1.24.0 - platform: local macOS arm64 machine
This is a useful first baseline, but it is still one machine, one target, and one cold-start-oriented scenario. The timings below should be treated as representative local data, not universal constants.
For the current state summary that ties these measurements to the architecture and the kept or reverted experiments, see go_orchestrion_maintainer_state.md.
After the later stdlib experiments and rollbacks, the branch was re-measured from the current good state using three runs in the sibling fixture repo:
- cold build:
fresh
--output_baseplus isolatedXDG_CACHE_HOME - warm build:
fresh
--output_basewhile reusing that same cache root - runtime validation:
fresh test execution with
DD_TRACE_DEBUG=1andDD_CIVISIBILITY_ENABLED=1
Measured result:
- cold build:
- total elapsed:
249.053s - critical path:
75.20s extensions.orchestrion_build_total:153.057sextensions.download_and_extract:32.066sextensions.go_build:113.334s
- total elapsed:
- warm build:
- total elapsed:
86.281s - critical path:
68.91s extensions.orchestrion_build_total:3.955s
- total elapsed:
- runtime validation:
- total elapsed:
96.620s - critical path:
81.32s extensions.orchestrion_build_total:3.532s
- total elapsed:
Runtime validation also confirmed that the measured branch behaved correctly:
- tracer startup logs were present
- CI Visibility initialized
- the test binary reported
Datadog Tracer v2.7.0-dev.1 - the test wrote a payload file under
test.outputs/payloads/tests
This latest baseline keeps the same broad conclusion as the earlier measurements:
- on a true cold start, the Orchestrion tool build is still the largest remaining cost
- once bootstrap reuse kicks in, stdlib becomes the most visible remaining build step
Two probe paths were involved:
- extension probes:
enabled with
log_timing = Truein the localorchestrion.from_source(...)call in the fixture repo - builder probes:
enabled with
RULES_GO_ORCHESTRION_PROBE=1
The builder probe plumbing also needed one fix before the measurements were useful:
- the compile, link, and stdlib actions were constructing explicit action environments and were not forwarding the probe environment variables into the builder process
- after forwarding those variables, successful actions started surfacing probe lines in normal Bazel output and mirroring them into a shared probe file
GIT_CONFIG_GLOBAL=/dev/null \
GIT_TERMINAL_PROMPT=0 \
PATH=$HOME/sdk/go1.24.0/bin:$PATH \
RULES_GO_ORCHESTRION_PROBE=1 \
./bazelw \
--output_base=/tmp/rto_phase12_cold/output_base \
test //src/go-project:hello_test \
--test_output=streamed \
--sandbox_debug \
--subcommands \
--verbose_failures \
--action_env=RULES_GO_ORCHESTRION_PROBE=1 \
--action_env=RULES_GO_ORCHESTRION_PROBE_FILE=/tmp/rto_phase12_cold/builder-probes.logThis run populated the shared helper-decision cache and the shared module-export cache using the Phase 1/2 code.
GIT_CONFIG_GLOBAL=/dev/null \
GIT_TERMINAL_PROMPT=0 \
PATH=$HOME/sdk/go1.24.0/bin:$PATH \
RULES_GO_ORCHESTRION_PROBE=1 \
./bazelw \
--output_base=/tmp/rto_phase12_post_export_seed/output_base \
test //src/go-project:hello_test \
--test_output=streamed \
--sandbox_debug \
--subcommands \
--verbose_failures \
--action_env=RULES_GO_ORCHESTRION_PROBE=1 \
--action_env=RULES_GO_ORCHESTRION_PROBE_FILE=/tmp/rto_phase12_post_export_seed/builder-probes.logThis is the important verification shape for the first implementation pass:
- keep the decision cache and shared module-export cache warm
- delete only the whole helper-archive bundle
- rerun with a fresh output base
- confirm that helper selection and export resolution are reused on the forced helper rebuild path
rm -rf \
/var/folders/42/00hrt9gj3276cy8h0_d9nw480000gq/T/datadog-orchestrion-go-cache/cache/rules-go-orchestrion/synthetic-testmain-helpers/5c8037544d53d7ef
GIT_CONFIG_GLOBAL=/dev/null \
GIT_TERMINAL_PROMPT=0 \
PATH=$HOME/sdk/go1.24.0/bin:$PATH \
RULES_GO_ORCHESTRION_PROBE=1 \
./bazelw \
--output_base=/tmp/rto_phase12_post_export_verify/output_base \
test //src/go-project:hello_test \
--test_output=streamed \
--sandbox_debug \
--subcommands \
--verbose_failures \
--action_env=RULES_GO_ORCHESTRION_PROBE=1 \
--action_env=RULES_GO_ORCHESTRION_PROBE_FILE=/tmp/rto_phase12_post_export_verify/builder-probes.logThe builder now mirrors probe lines into the shared Orchestrion cache root.
On this machine, the measured run wrote to:
/var/folders/42/00hrt9gj3276cy8h0_d9nw480000gq/T/datadog-orchestrion-go-cache/probes/builder-probes.log
That exact base path is machine-specific. The important invariant is that the
file lives under the shared datadog-orchestrion-go-cache/probes/ directory.
The baseline run and the Phase 1/2 verification run both completed successfully.
Key top-level numbers from the measured runs:
- baseline cold run:
430.571selapsed,219.60scritical path - Phase 1/2 seed run:
334.254selapsed,169.85scritical path - Phase 1/2 verification run with helper archive removed:
341.734selapsed,163.97scritical path
The phase timings below are the meaningful comparison point. The Phase 1/2 verification run intentionally forced a helper-archive rebuild, but it kept the new lower-layer caches available so their reuse behavior could be measured.
From the cold extension setup pass:
extensions.validate_dd_trace_go_versions: about0.08sextensions.download_and_extract: about21-22sextensions.go_mod_edit: about8-9sextensions.go_mod_tidy: about96-104sextensions.go_build: about32-41sextensions.orchestrion_build_total: about160-176s
The biggest startup cost before normal Bazel Go actions even begin is the temporary Orchestrion module preparation and binary build.
In that earlier baseline, the largest single measured phase in that area was
go mod tidy.
This section records the intermediate bootstrap-cache experiment that preceded the later no-tool-repin design.
The next optimization pass focused only on the Orchestrion bootstrap path in
go/private/orchestrion/extensions.bzl.
This pass added:
- a host-side bootstrap artifact cache for the built Orchestrion binary and its
dd_trace_go_versions.json - stable host-side
GOMODCACHEandGOCACHEroots under the shareddatadog-orchestrion-go-cache - a bootstrap cache key that includes:
- Orchestrion version
- normalized
dd_trace_go_versions - host Go identity
- cache ABI version
- manual patchset identifier
- a fast path that reuses the cached bootstrap artifact on a cache hit
- a fallback path that still runs
go mod tidyonly when the first build says the module graph is not ready - Starlark unit tests for the retry classifier and bootstrap cache-key stability
Cold verification:
BASE=/tmp/rto_bootstrap_eval7
rm -rf "$BASE"
mkdir -p "$BASE/cold" "$BASE/warm" "$BASE/cache_home"
cd ../rules_test_optimization_tests
source "$HOME/ddtrace.sh" >/dev/null 2>&1 || true
PATH=$HOME/sdk/go1.24.0/bin:$PATH \
RULES_GO_ORCHESTRION_PROBE=1 \
XDG_CACHE_HOME="$BASE/cache_home" \
RULES_GO_ORCHESTRION_PROBE_FILE="$BASE/cold/builder-probes.log" \
./bazelw \
--output_base="$BASE/cold/output_base" \
build //src/go-project:hello_test \
--subcommands \
--verbose_failures \
>"$BASE/cold/run.log" 2>&1Warm verification:
cd ../rules_test_optimization_tests
source "$HOME/ddtrace.sh" >/dev/null 2>&1 || true
PATH=$HOME/sdk/go1.24.0/bin:$PATH \
RULES_GO_ORCHESTRION_PROBE=1 \
XDG_CACHE_HOME=/tmp/rto_bootstrap_eval7/cache_home \
RULES_GO_ORCHESTRION_PROBE_FILE=/tmp/rto_bootstrap_eval7/warm/builder-probes.log \
./bazelw \
--output_base=/tmp/rto_bootstrap_eval7/warm/output_base \
build //src/go-project:hello_test \
--subcommands \
--verbose_failures \
>/tmp/rto_bootstrap_eval7/warm/run.log 2>&1The best measured variant at that stage kept three things:
- the host-side bootstrap artifact cache
- the persistent host-side Go caches
go mod tidyas a fallback instead of as an unconditional step
Measured result from that intermediate variant:
- cold bootstrap:
extensions.download_and_extract:21.907sextensions.go_mod_edit:9.831sextensions.go_mod_download:53.575sextensions.go_build_initial:55.091swith fallbackextensions.go_mod_tidy:30.131sextensions.go_build_retry:37.981s
One later stdlib experiment looked good from the build timings and was pushed, but it had to be rolled back.
The host-side woven stdlib snapshot cache made build/test timing look better, but it silently broke runtime weaving:
- Bazel builds still succeeded
- Bazel tests still passed
- the test binary still contained
dd-trace-go - but CI Visibility never started at runtime
- tracer startup logs disappeared
- payload files were not written
- the uploader then had nothing to upload
This failure reproduced both in CI and locally.
For stdlib-path changes, build success is not enough.
Any future stdlib optimization must also be validated in
../rules_test_optimization_tests with:
DD_TRACE_DEBUG=1DD_CIVISIBILITY_ENABLED=1- confirmation that the tracer startup logs appear
- confirmation that payload files are written under
test.outputs/payloads
If those runtime signals are missing, the optimization is not safe even if the build and test steps are green.
extensions.orchestrion_build_total:216.628s- end-to-end build:
305.653selapsed - warm bootstrap:
extensions.bootstrap_cache_hit: yesextensions.orchestrion_build_total:3.097s- no
download_and_extract,go_mod_edit,go_mod_download,go_mod_tidy, orgo_buildphases executed on the hit - end-to-end build:
91.957selapsed
This pass clearly improved warm bootstrap reuse. The host-side bootstrap cache removes almost all of the repeated Orchestrion setup cost across fresh Bazel output bases.
It did not achieve the hoped-for cold-start reduction yet. Several follow-up experiments were measured locally, including:
- using
go build -mod=modon the first build - skipping the explicit tracer-module download
Those experiments made the cold bootstrap slower on this machine, so they were not kept.
The practical outcome is:
- warm bootstrap reuse is now real and large
- cold bootstrap is still expensive
- the next extension-focused pass should look for a better way to prepare the Orchestrion module graph without paying for both a large module download and a later rebuild
The next pass removed the tool-side module rewrite entirely. Bazel still:
- downloads and patches the Orchestrion source tree
- writes
dd_trace_go_versions.json - validates the configured tracer version against the target module later in the builder/runtime path
But it no longer runs go mod edit, go mod download, or go mod tidy
against the downloaded Orchestrion repo itself.
Consumer validation:
cd ../rules_test_optimization_tests
eval "$(sed -n '/^export DD_API_KEY=/p;/^export DD_SITE=/p' ~/ddtrace.sh)"
TEST_TARGET=//src/go-project:hello_test \
DD_TRACE_DEBUG=1 \
./runtests
TEST_TARGET=//src/go-project:hello_test \
DD_TRACE_DEBUG=1 \
./runtests-hermeticTracer log validation:
cd ../rules_test_optimization_tests
eval "$(sed -n '/^export DD_API_KEY=/p;/^export DD_SITE=/p' ~/ddtrace.sh)"
TEST_TARGET=//src/go-project:hello_test \
DD_TRACE_DEBUG=1 \
DD_CIVISIBILITY_ENABLED=1 \
./runtestsCold bootstrap with the no-tool-repin flow:
extensions.validate_dd_trace_go_versions:87 msextensions.download_and_extract:22.192 sextensions.patch_source_tree:61 msextensions.go_build_initial:25.886 sextensions.bootstrap_cache_write:430 msextensions.orchestrion_build_total:49.742 s
Removed phases on that cold run:
extensions.go_mod_edit:0extensions.go_mod_download:0extensions.go_mod_tidy:0extensions.go_mod_tidy_fallback:0
Comparison to the earlier cold bootstrap baseline:
- before: about
175.2 s - after: about
49.7 s - improvement: about
125.5 sfaster, roughly71.6%lower
Target-side verification:
./runtestspassed./runtests-hermeticpassed- with
DD_CIVISIBILITY_ENABLED=1, the tracer logs showedDatadog Tracer v2.7.0-dev.1, confirming that the pinned target tracer version was still the one loaded by the test binary
The first implementation pass used these measured gates:
compilepkg.compile_synthetic_testmain_source_packagesimportcfg.resolve_module_exports_for_packages.go_list_export_depsbuilder.stdlibextensions.go_mod_tidy
The target outcomes for this pass were:
- synthetic testmain cold path: major reduction
- export resolution: major reduction
- stdlib: moderate improvement if the cache-key cleanup also helped there
- extension bootstrap: observe, but do not optimize yet
The Phase 1/2 verification builder probe log contained 3550 structured
timing lines.
builder.compilepkgfor the synthetic testmain action:134179 mscompilepkg.compile_synthetic_testmain_source_packages:132384 msimportcfg.resolve_module_exports_for_packages.go_list_export_deps:18848 msbuilder.stdlib:52907 msextensions.go_mod_tidy:127946 ms
builder.compilepkgfor the synthetic testmain action:81538 mscompilepkg.compile_synthetic_testmain_source_packages:79321 msbuilder.link:3582 msbuilder.stdlib:45097 msextensions.go_mod_tidy:98580 ms
The synthetic testmain path is not slow because the final compile command is slow. It is slow because a large amount of helper-package preparation happens before the final compile is attempted.
Important measured phases from the testmain action:
compilepkg.synthetic_testmain_helper_cache_miss: intentionally forced by deleting only the helper-archive bundlecompilepkg.synthetic_testmain_helper_decision_cache_hit: confirmed reuse of the persisted helper decision graphimportcfg.resolve_module_exports_for_packages.cache_hit: confirmed reuse of the shared module-export cacheimportcfg.resolve_module_exports_for_packages.go_list_export_deps: not executed on the verification runcompilepkg.compile_synthetic_testmain_source_packages:79321 ms- final
compilepkg.compile_go_action.run_commandfor the main synthetictestmainpackage:1343 ms
The builder-side bottleneck is dominated by synthetic testmain helper preparation, not by the final compile or link command.
The stdlib path is also expensive, but it is clearly smaller than the synthetic testmain helper path in this measured scenario.
The important change from this first optimization pass is that the expensive export-discovery work is no longer being repeated during a forced helper-bundle rebuild. The remaining cost is mostly real source compilation for the selected helper closure.
The first implementation pass did three concrete things:
- It made the cache keys stable across fresh Bazel output bases.
- It persisted the synthetic helper decision graph separately from the whole helper-archive bundle.
- It persisted the resolved module-export map under a shared request-keyed
cache so equivalent helper rebuilds can skip
go list -export -deps.
The measured result is:
- synthetic helper source-package time dropped from
132.4sto79.3son the forced helper rebuild path - that is about a
40.1%reduction - synthetic testmain
builder.compilepkgtime dropped from134.2sto81.5s - that is about a
39.2%reduction importcfg.resolve_module_exports_for_packages.go_list_export_depswent from18.8sin the baseline run to not executing at all in the verification run because the shared export cache hitbuilder.stdlibimproved from52.9sto45.1sin the verification run, which is about a14.8%reduction
These numbers do not mean the whole build is solved. They do show that the Phase 1/2 cache reuse is real and materially changes the synthetic helper rebuild path.
This section is partly historical.
It records the priority order as it looked immediately after the Phase 1/2
synthetic-helper cache work, before the later bootstrap redesign removed the
tool-side go mod tidy path. For the current maintained summary, prefer
go_orchestrion_maintainer_state.md.
Before measuring, the likely hot paths were:
- Orchestrion bootstrap/setup
- stdlib weaving
- synthetic testmain handling
- repeated importcfg rewrites and export discovery
After the baseline pass, the priority order was:
- synthetic testmain helper preparation
- Orchestrion extension
go mod tidy - stdlib install plus export persistence/sync
- repeated export discovery through
go list -export -deps
After the first implementation pass, the priority order was:
- remaining synthetic helper source compilation
- Orchestrion extension
go mod tidy - stdlib install plus export persistence/sync
- repeated woven dependency probing
That means the next pass should continue on the synthetic helper source compilation logic itself, not go back to export resolution for this path.
That priority list is only accurate for the pre-no-tool-repin phase of the
investigation. Later work removed the extension-side go mod tidy path, so it
is no longer a current hotspot on this branch.
Because the synthetic testmain path does not stop at the small fixed Datadog root helper set.
The flow in compilepkg.go starts from:
syntheticTestmainRootPackagesorchestrionLinkClosurePackages
Then compileSyntheticTestmainSourcePackages(...) expands that into a larger
source-compiled closure by:
- loading dependency metadata with
go list -deps -json - recursively classifying packages in
packageNeedsSyntheticSourceCompile(...) - recursively compiling selected packages in
compileSyntheticTestmainSourcePackage(...)
The recursive selection rule is the key detail. A package is pulled into source compilation when it:
- is already in the root set
- depends on another package that needs source compilation
- or imports certain stdlib packages such as
flag,log,log/slog,net/http,os,os/exec, ortesting
So the cold miss is not “compile a few helper packages”. It becomes “compile a large transitive non-cgo helper ecosystem that must remain source-compatible in the synthetic module”.
That matches the measured run: the expensive packages were not only the obvious Datadog roots. They also included deeper transitive packages such as:
github.com/DataDog/dd-trace-go/v2/ddtrace/mocktracergithub.com/DataDog/datadog-agent/pkg/trace/statsgoogle.golang.org/protobuf/runtime/protoimpl
Some reuse already exists, but it is coarse.
The current persistent helper cache in
compileSyntheticTestmainSourcePackages(...) already stores:
- compiled helper interface archives
- compiled helper link archives
- helper link closures
That cache is shared across similar actions because its key depends on things like:
- configured
dd-trace-goversions - SDK path
- install suffix
- woven stdlib key
- Orchestrion version
What is reused more aggressively after Phase 1/2:
- package metadata from
loadModulePackageMetadataBatch(...) - recursive source-compile decisions from
packageNeedsSyntheticSourceCompile(...) - external export resolution done during a helper-cache miss
- the whole helper bundle when its top-level cache key matches
What is still not reused aggressively enough today:
- individual helper package compile results outside the whole-bundle helper cache
- the remaining real source compilation work inside a forced helper-bundle rebuild
So the strongest reuse opportunities are:
- consider finer-grained per-helper-package reuse if the coarse whole-bundle cache is still too expensive on misses
- reduce how many packages need to be source-compiled in the first place
Yes.
The code already has most of the right key material through
moduleExportRequestKey(...) in
importcfg.go.
That key already includes:
- helper export cache ABI version
- Orchestrion version identity
- SDK path
- install suffix
- whether the module root is synthetic or real
- digests of
go.mod,go.sum,orchestrion.tool.go, andorchestrion.yml - a woven stdlib cache key
Before this pass, the synthetic cold-miss path did not reuse that aggressively
because it called resolveModuleExportsForPackagesWithRoot(...) with a forced
temporary export root under the helper-cache temp directory. That bypassed the
stable shared module-exports/<requestKey> cache root used by the normal path.
Yes, and this pass now proves it in the measured fixture run.
The verification run intentionally removed only the whole helper-archive bundle and kept the lower-layer caches warm. The resulting probe lines showed:
- helper archive cache miss
- helper decision cache hit
- module export cache hit
- no
go list -export -depsexecution forresolve_module_exports_for_packages
So the direct answer is no longer theoretical. The result now survives across similar synthetic testmain actions when the shared cache key matches.
This was the next ranked plan at that point in the investigation. Later work on this branch changed the bootstrap design substantially, so the current next steps are different from the list below.
This is the clearest builder hotspot.
Specific focus areas now:
- reduce the remaining real source compilation inside
compilepkg.compile_synthetic_testmain_source_packages - identify whether some helper packages can be precompiled or reused more granularly without changing behavior
- investigate whether the selected helper closure can be narrowed safely for common testmain shapes
At that time, this was the largest measured startup phase before the Go actions even began.
Specific focus areas:
- understand why
go mod tidydoes so much Git/network work on cold setup - determine whether the temporary module can be prepared with less work than a full tidy
- investigate whether a more reusable cache layout can avoid paying this cost repeatedly in local runs
This item is historical. The later no-tool-repin change removed that extension-side module rewrite and tidy path from the current design.
The stdlib action is not just expensive because of installation. A large amount of time is spent persisting and copying export data after the install.
Specific focus areas:
stdlib.persist_orchestrion_stdlib_exportsstdlib.sync_persisted_exports_to_cacheimportcfg.resolve_cache_stdlib_exports_at.go_list_export_deps
Multiple phases repeatedly run ensure_woven_packages_available and related
go list probes.
The per-call cost is smaller than the biggest hotspots, but it appears in many places and may add up significantly across larger builds.
The first implementation pass was verified with:
- the baseline fixture run in
../rules_test_optimization_tests - a seed run that populated the new caches
- a verification run that deleted only the helper-archive bundle and confirmed lower-layer cache reuse
- focused builder tests in this repo:
./bazelw test @rules_go//go/tools/builders:{compilepkg_test,importcfg_test,probe_test,orchestrion_test}
All of those checks passed locally.
These were the practical next steps before the later bootstrap redesign.
The next optimization pass should start with the synthetic testmain path in:
go/tools/builders/compilepkg.gogo/tools/builders/importcfg.go
The first concrete questions to answer are:
- why are so many helper packages being compiled on a cold miss?
- which of those helper package results can be reused more aggressively?
- can the
resolve_module_exports_for_packagesresult be cached in a way that survives across similar testmain actions?
The second optimization pass should then focus on:
go/private/orchestrion/extensions.bzlgo/tools/builders/stdlib.gogo/tools/builders/orchestrion.go
- docs/rules_go_orchestrion_performance_analysis.md
- third_party/rgo/v0_60_0/base/go/tools/builders/compilepkg.go
- third_party/rgo/v0_60_0/base/go/tools/builders/importcfg.go
- third_party/rgo/v0_60_0/base/go/tools/builders/stdlib.go
- third_party/rgo/v0_60_0/base/go/tools/builders/orchestrion.go
- third_party/rgo/v0_60_0/base/go/private/orchestrion/extensions.bzl