Skip to content

Fix Path issues for Slurm clusters and handle non-standard installation paths. #6982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
copybara-service wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fix Path issues for Slurm clusters and handle non-standard installation paths. #6982

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions perfkitbenchmarker/cluster.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,12 +247,14 @@ def RemoteCommand(

Returns:
A tuple of stdout and stderr from running the command.

Raises:
RemoteCommandError: If there was a problem issuing the command.
"""
slurm_path = (
'export PATH=$PATH:/usr/local/bin:/usr/local/sbin:'
'/apps/slurm/current/bin:/apps/slurm/bin:'
'/opt/slurm/bin:/usr/local/slurm/bin; '
)
return self.headnode_vm.RemoteCommand(
f'{env} srun -N {self.num_workers} {command}',
f'{slurm_path}{env} srun -N {self.num_workers} {command}',
ignore_failure=ignore_failure,
timeout=timeout,
**kwargs,
Expand Down Expand Up @@ -372,14 +374,19 @@ def Delete(self):
@vm_util.Retry(
fuzz=0,
timeout=1800,
max_retries=5,
max_retries=30,
retryable_exceptions=(errors.Resource.RetryableCreationError,),
)
def _WaitForClusterReady(self):
if self.unmanaged:
return
slurm_path = (
'export PATH=$PATH:/usr/local/bin:/usr/local/sbin:'
'/apps/slurm/current/bin:/apps/slurm/bin:'
'/opt/slurm/bin:/usr/local/slurm/bin; '
)
if not self.headnode_vm.TryRemoteCommand(
f'srun -N {self.num_workers} hostname'
f'{slurm_path}srun -N {self.num_workers} hostname'
):
raise errors.Resource.RetryableCreationError('Cluster not ready.')

Expand Down
34 changes: 33 additions & 1 deletion perfkitbenchmarker/data/cluster/cluster_toolkit.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,35 @@ vars:
region: {{ region }}
zone: {{ zone }}

{% if 'staging' in zone %}
# Add skip validators for staging
validators:
- {validator: test_zone_exists, skip: true}
- {validator: test_zone_in_region, skip: true}
- {validator: test_region_exists, skip: true}
- {validator: test_machine_type_in_zone, skip: true}
- {validator: test_disk_type_in_zone, skip: true}

# Add provider overrides for staging
terraform_providers:
google:
source: hashicorp/google
version: ">= 6.22.0"
configuration:
project: $(vars.project_id)
region: $(vars.region)
zone: $(vars.zone)
compute_custom_endpoint: "https://compute.mtls.googleapis.com/compute/staging_v1/"
google-beta:
source: hashicorp/google-beta
version: ">= 6.22.0"
configuration:
project: $(vars.project_id)
region: $(vars.region)
zone: $(vars.zone)
compute_custom_endpoint: "https://compute.mtls.googleapis.com/compute/staging_v1/"
{% endif %}

deployment_groups:
- group: primary
modules:
Expand All @@ -32,7 +61,10 @@ deployment_groups:
enable_public_ips: true
enable_oslogin: false
machine_type: {{ worker_machine_type }}
disk_type: pd-balanced
disk_type: {{ 'hyperdisk-balanced' if 'h4d' in worker_machine_type else 'pd-balanced' }}
{% if 'h4d' in worker_machine_type %}
on_host_maintenance: TERMINATE
{% endif %}
bandwidth_tier: gvnic_enabled
allow_automatic_updates: false
instance_image_custom: true
Expand Down