Variant Configs & Sweep Tools - #6901
Conversation
3b6a9ef to
9345124
Compare
|
/cc @pmandewalkar |
| return sorted(all_metrics) | ||
|
|
||
|
|
||
| def get_median(values): |
There was a problem hiding this comment.
nit: i see that in sweep.py we import statistics and use that median, is there a reason why that's being redefined here?
There was a problem hiding this comment.
removed the custom get_median function and updated the script to use Python's built-in statistics.median().
| f"kubectl delete sandboxclaims --all -n {namespace} --ignore-not-found=true", | ||
| check=False, | ||
| ) | ||
| for _ in range(100): |
There was a problem hiding this comment.
Can we use the built in kubernetes wait function here?
There was a problem hiding this comment.
done, kubectl wait --for=delete.
pmandewalkar
left a comment
pmandewalkar
left a comment
There was a problem hiding this comment.
I see a good amount of YAML duplication across configs like baseline.yaml, c4a_arm.yaml, c4d_amd.yaml, larger_vm.yaml, etc., with mostly identical benchmark definitions. I think for overall easier maintenance and longevity could we use templating similar to how it is done in sched_tuner_daemonset.yaml.j2?
5d0ef01 to
92472bb
Compare
@pmandewalkar PKB's |
| - | | ||
| echo 2212926 > /proc/sys/kernel/sched_migration_cost_ns | ||
| echo 4656091 > /proc/sys/kernel/sched_min_granularity_ns | ||
| echo 26403982 > /proc/sys/kernel/sched_wakeup_granularity_ns |
There was a problem hiding this comment.
Can you document what are the defaults?
| swapConfig: | ||
| enabled: true | ||
| sysctl: | ||
| vm.swappiness: '15' |
There was a problem hiding this comment.
Was kubelet swap configuration missing during the experiment?
https://kubernetes.io/docs/tutorials/cluster-management/provision-swap-memory/#set-up-kubelet-configuration
There was a problem hiding this comment.
yes my understanding is that since we are using the --system-config-from-file to set swapConfig.enabled: true, GKE automatically handles the underlying Kubelet configuration (i.e setting failSwapOn=false and swapBehavior=LimitedSwap on the nodes)
| swapConfig: | ||
| enabled: true | ||
| sysctl: | ||
| vm.swappiness: '15' |
There was a problem hiding this comment.
There is a kubernetes blog post on recommended swap configuration: https://kubernetes.io/blog/2025/08/19/tuning-linux-swap-for-kubernetes-a-deep-dive/#recommended-starting-point
There was a problem hiding this comment.
The results ultimately proved that our test workloads are almost strictly CPU-bound (hitting 95%+ CPU PSI), while memory remained highly stable, meaning fine-tuning the swappiness further wouldn't have shifted the saturation ceiling.
c28d877 to
fca492f
Compare
3 participants
Files: ~20 new files
perfkitbenchmarker/data/k8s_agents/config/agentic_benchmark_config.yamlperfkitbenchmarker/data/k8s_agents/config/variants/baseline.yamlperfkitbenchmarker/data/k8s_agents/config/variants/kubelet_pulls.yamlperfkitbenchmarker/data/k8s_agents/config/variants/overlay_none.yamlperfkitbenchmarker/data/k8s_agents/config/variants/larger_vm.yamlperfkitbenchmarker/data/k8s_agents/config/variants/hyperdisk_200gb.yamlperfkitbenchmarker/data/k8s_agents/config/variants/thp.yamlperfkitbenchmarker/data/k8s_agents/config/variants/sched_tuning.yamlperfkitbenchmarker/data/k8s_agents/config/variants/swap.yamlperfkitbenchmarker/data/k8s_agents/config/variants/c4d_amd.yamlperfkitbenchmarker/data/k8s_agents/config/variants/c4a_arm.yamlperfkitbenchmarker/data/k8s_agents/config/variants/multi_node.yamlperfkitbenchmarker/data/k8s_agents/config/variants/variant_reference_guide.mdperfkitbenchmarker/data/k8s_agents/config/variants/tuning/kubelet_parallel_pulls.yamlperfkitbenchmarker/data/k8s_agents/config/variants/tuning/sched_tuner_daemonset.yaml.j2perfkitbenchmarker/data/k8s_agents/config/variants/tuning/swap_config.yamlperfkitbenchmarker/data/k8s_agents/config/variants/tuning/thp_config.yamlperfkitbenchmarker/scripts/agentic/sweep.pyperfkitbenchmarker/scripts/agentic/analyze.py.gitignoreadditionsDescription: Adds optimization sweep infrastructure:
k8s_snapshotkey including--enable-pod-snapshotsingke_additional_flags.agentic_benchmark_config.yaml) with YAML anchors for consistent cluster definitionssweep.py) — orchestrates multi-variant benchmark sweeps across all 7 benchmarks with warm pool drain between sweep levelsanalyze.py) — parses PKB NDJSON results, computes deltas vs baseline, generates markdown reports and CSV exports. Supports all 7 benchmarks via registry with auto-discovery fallback.