Shared Utilities & Infrastructure by george-kalisse-sada · Pull Request #6896 · GoogleCloudPlatform/PerfKitBenchmarker

george-kalisse-sada · 2026-07-17T03:14:53Z

Files: ~10 new files

perfkitbenchmarker/linux_benchmarks/kubernetes/__init__.py
perfkitbenchmarker/linux_benchmarks/kubernetes/agentic/__init__.py
perfkitbenchmarker/linux_benchmarks/kubernetes/agentic/README.md
perfkitbenchmarker/linux_benchmarks/kubernetes/agentic/k8s_benchmark_utils.py
perfkitbenchmarker/linux_benchmarks/kubernetes/agentic/gke_deploy_utils.py
perfkitbenchmarker/scripts/agentic/__init__.py
perfkitbenchmarker/scripts/agentic/README.md
perfkitbenchmarker/scripts/agentic/gke_image_build_utils.py
perfkitbenchmarker/scripts/agentic/gke_prerequisites.py
perfkitbenchmarker/scripts/agentic/gke_post_teardown.py

Description: Adds shared infrastructure for GKE Agent Sandbox benchmarks:

k8s_benchmark_utils.py — Agent API interaction, kubectl helpers, warm pool management, port-forward manager, PKB sample construction
gke_deploy_utils.py — Idempotent deployment of Agent Sandbox ecosystem (CRDs, SandboxTemplates, WarmPools, Router, ADK Agent) onto pre-provisioned GKE clusters
gke_image_build_utils.py — Container image builds (Chrome Sandbox, Sandbox Router) via Google Cloud Build with cross-architecture support (amd64/arm64)
gke_prerequisites.py — One-time GCP project setup (APIs, Artifact Registry, Cloud Build SA, image builds including native ARM64 VM builds)
gke_post_teardown.py — Cleanup of resources created by prerequisites

hubatish · 2026-08-05T18:30:12Z

/gcbrun

roycaihw · 2026-08-06T15:56:51Z

/cc @Nishanth29

Nishanth29 · 2026-08-06T16:57:52Z

Hi @george-kalisse-sada , it looks like the CI check Pull (bionic-baton-343) failed during execution. could you check the build failure and rerun/resolve the CI run so all checks pass?

Nishanth29 · 2026-08-07T17:27:27Z

Per @george-kalisse-sada, the pull failure was likely a transient timeout issue. Zach synced to head and re-ran the check, which passed successfully(documenting here for visibility).

Nishanth29 · 2026-08-07T17:44:07Z

+    """Run a shell command, raising on failure."""
+    logger.info("  CMD: %s", " ".join(cmd))
+    env = os.environ.copy()
+    env["CLOUDSDK_AUTH_DISABLE_SSL_VALIDATION"] = "true"


Is CLOUDSDK_AUTH_DISABLE_SSL_VALIDATION needed here or was it left over from local testing? Can we drop this line?

Nishanth29 · 2026-08-07T17:59:26Z

+    rendered = template.render(**kwargs)
+
+    # Write rendered YAML to tmp dir (RunKubectlCommand does not support stdin)
+    tmp_dir = os.path.join(


Doesn't writing rendered manifests to data.ResourcePath(...) creates temp files inside the repository source tree?? ig we use PKB's managed temp dir instead so it doesn't mess git or fail in read-only installs.

Yes, changed to use PKB's managed vm_util.GetTempDir()

Nishanth29 · 2026-08-07T18:03:41Z

+import threading
+
+
+_PID_FILE = "/tmp/pkb_portforward.pid"


Wondering if two sweeps running in parallel might overwrite this /tmp file? Maybe we can put it in PKB's temp directory or add the port to the filename.

updated the PID file path to dynamically include the local port number (e.g., pkb_portforward_8080.pid).

roycaihw · 2026-08-14T00:00:22Z

@@ -0,0 +1,54 @@
+# GKE Agent Sandbox Benchmarks


Replace "GKE" with "Kubernetes". Please also update other places that use "GKE"

roycaihw · 2026-08-14T00:01:58Z

+
+PKB benchmark modules for measuring GKE Agent Sandbox performance under
+gVisor isolation. Each benchmark is an atomic single-point measurement
+designed to be invoked repeatedly by the sweep runner with varying parameters.


Each benchmark is a standalone PKB benchmark.

Updated to 'Each benchmark is a standalone PKB benchmark.' for clarity.

roycaihw · 2026-08-14T00:02:31Z

+| Python Density | `k8s_python_density_benchmark.py` | Yes | CEL, TTFE, RSS, per-type latency (compute/syscall/import) |
+| Payload Transfer | `k8s_payload_benchmark.py` | Yes | Generation, serialization, stdout write, throughput |
+| Chromium Density | `k8s_chromium_density_benchmark.py` | Yes | Navigate, evaluate, click, fill, screenshot latency |
+| QPS Saturation | `k8s_qps_benchmark.py` | Yes/No | TTFE at controlled request rates, warm pool drain detection |


What does "Agent API: Yes/No" mean?

Removed, It was originally intended to indicate whether the benchmark interacted with the custom the Agent's FastAPI service or just the raw K8s API, but yea it's not necessary for the documentation and creates confusion.

roycaihw · 2026-08-14T00:03:45Z

+flags.DEFINE_string(
+    "k8s_agentic_benchmark_note",
+    "",
+    "Arbitrary note string attached to every sample for tagging runs.",


Please elaborate. What does note mean here and what's is for?

I've expanded the help string. It's used to inject a custom metadata tag (like 'nightly_run_v2') into the emitted samples so they can be easily filtered and grouped in downstream dashboards if needed.

roycaihw · 2026-08-14T00:06:17Z

+    Delegates to PKB's native kubectl module which handles kubeconfig
+    and retries for transient connection errors automatically.
+    """
+    return kubectl.RunKubectlCommand(


Why do we need to duplicate this? This method adds no value but complexity.

Gentle ping. Let's remove this wrapper.

I forgot to reply back to this before but below would be my response:

The reason I kept this is because I wanted a global consistent timeout and raise_on_failure config for all benchmarks, rather than specifying them everytime I call the function. i would only specify them where they are not the default I chose in the wrapper.

During our saturation sweeps, if the K8s API server becomes unresponsive, we need it to fail fast rather than hanging the PKB thread indefinitely.

roycaihw · 2026-08-14T00:17:26Z

+            ]
+        )
+        _RunCmd(
+            ["git", "sparse-checkout", "set", "examples/chrome-sandbox"],


Are we simply building example code from https://github.com/kubernetes-sigs/agent-sandbox? Ideally this should be built by the agent-sandbox repo so we can simply reuse published images.

Ideally yes, we would use the published images. However, we had to build them from source for two reasons:

We needed to patch the Chrome Dockerfile to inject socat for the CDP proxy.

The upstream repository does not currently publish arm64 images, which we need for the ARM saturation sweeps.

Resolving for now. I'm tracking the followup items and will open OSS issues / track internally.

roycaihw · 2026-08-14T00:17:49Z

+                "git",
+                "sparse-checkout",
+                "set",
+                "clients/python/agentic-sandbox-client/sandbox-router",


Same here. Are we simply building example code from https://github.com/kubernetes-sigs/agent-sandbox? Ideally this should be built by the agent-sandbox repo so we can simply reuse published images.

The upstream repository does not currently publish arm64 images, which we need for the ARM saturation sweeps.

Resolving for now. I'm tracking the followup items and will open OSS issues / track internally.

roycaihw · 2026-08-14T00:18:32Z

@@ -0,0 +1,74 @@
+# GKE Agentic Benchmark Scripts


Replace "GKE" with "Kubernetes". You should scan all your code changes, find places that use "GKE" and update accordingly

Done, but it's not a replace all exercise, since there are some things that are GKE specific. I might have missed a few things before, but I probably caught all now.

roycaihw · 2026-08-14T00:20:35Z

+
+### Sweep Runner
+
+`sweep.py` orchestrates multi-variant benchmark sweeps. It calls PKB


I don't see sweep.py in this PR. Ideally we should remove this paragraph from this PR. We can discuss whether sweep.py belongs to PKB repo in a separate PR.

The PR strategy document we discussed had splitting the mega-PR by whole files rather than fragmenting file contents across multiple PRs. So, this README describes the final state of the scripts/agentic directory. Content fragmentation at the stage we decided into split PRs would have've been prone to errors.

roycaihw · 2026-08-14T00:23:45Z

+    logger.info("Snapshot bucket deleted.")
+
+
+def teardown_images(project_id: str, region: str) -> None:


Do these teardown operations happen on-demand, or do they happen during each benchmark? I'd expect it to be the former (otherwise it will be very time-consuming to do the unnecessary re-build), but just want to confirm with you.

Yes, It is the former. These teardown operations (like deleting the Artifact Registry and GCS buckets) do not happen during each benchmark. I've clarified this in the script's docstring.

roycaihw

These two comments are still not addressed. Please address them

george-kalisse-sada force-pushed the SADA_2_shared_utils branch from 9708030 to dc93fba Compare August 5, 2026 17:39

Nishanth29 reviewed Aug 7, 2026

View reviewed changes

roycaihw reviewed Aug 14, 2026

View reviewed changes

roycaihw reviewed Aug 26, 2026

View reviewed changes

Shared Utilities & Infrastructure

8c1a178

george-kalisse-sada force-pushed the SADA_2_shared_utils branch from 68520f6 to 8c1a178 Compare August 30, 2026 01:13


		### Sweep Runner

		`sweep.py` orchestrates multi-variant benchmark sweeps. It calls PKB

		logger.info("Snapshot bucket deleted.")


		def teardown_images(project_id: str, region: str) -> None:

		import threading


		_PID_FILE = "/tmp/pkb_portforward.pid"

Uh oh!

Conversation

george-kalisse-sada commented Jul 17, 2026

Uh oh!

hubatish commented Aug 5, 2026

Uh oh!

roycaihw commented Aug 6, 2026

Uh oh!

Nishanth29 commented Aug 6, 2026

Uh oh!

Nishanth29 commented Aug 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roycaihw left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!