Npi raid0 lssd throughput tests

[PranjalC100]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @PranjalC100, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a comprehensive benchmarking solution for Google Cloud Local NVMe SSDs configured in a RAID 0 array. It provides a new Docker image and an accompanying Python script to automate the entire process, from identifying available LSSDs and setting up the RAID 0 volume to executing FIO-based throughput tests and uploading the performance metrics to BigQuery. This enhancement allows for standardized and repeatable performance evaluation of LSSD configurations within the existing benchmarking framework.

Highlights

• New LSSD RAID 0 Throughput Benchmark: Introduces a dedicated benchmark for Google Cloud Local NVMe SSDs configured in a RAID 0 array to measure their raw throughput performance.
• Automated RAID 0 Setup and FIO Testing: A new Python script automates the discovery of Local NVMe SSDs, the creation of a RAID 0 array, formatting with ext4, mounting, and the execution of FIO benchmarks (sequential write/read, random read).
• BigQuery Integration for Results: Benchmark results, including bandwidth and IOPS, are automatically parsed from FIO output and uploaded to BigQuery for centralized analysis and tracking.
• Dedicated Docker Image for Benchmarking Environment: A new Dockerfile (lssd.dockerfile) is added to encapsulate all necessary system tools (mdadm, fio, xfsprogs, e2fsprogs, libaio1) and Python dependencies (google-cloud-bigquery) required for the LSSD benchmarking environment.
• Flexible Docker Command Execution: The npi.py script is enhanced to allow passing arbitrary extra_docker_flags to the docker run command, which is critically used here to mount the host's /dev directory into the benchmark container, enabling access to the physical NVMe devices.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new benchmark for testing LSSD throughput in a RAID 0 configuration. It includes a new Dockerfile, a Python script to manage the RAID array and run FIO tests, and modifications to the main NPI script and Cloud Build configuration to integrate the new benchmark. The changes are well-structured. I've provided a few suggestions to improve the robustness of the new benchmark script and to clean up the orchestrator script.

[gemini-code-assist]

The current method of creating the RAID array by piping 'y' to mdadm is fragile and can fail if mdadm has other prompts. A more robust approach is to first zero out the superblocks on the devices to prevent any interactive prompts, and then use subprocess.check_call for simpler and cleaner execution and error handling.

Suggested change

	try:
	process = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
	process.communicate(input=b'y\n')
	if process.returncode != 0:
	logger.error("Failed to create RAID array.")
	sys.exit(1)
	logger.info("RAID array created successfully.")
	except Exception as e:
	logger.error(f"Error during RAID creation: {str(e)}")
	sys.exit(1)
	# Zero out any old superblocks to avoid mdadm prompts.
	for device in devices:
	subprocess.run(['mdadm', '--zero-superblock', device], check=False, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
	try:
	# With superblocks zeroed, mdadm shouldn't need to prompt.
	# Using check_call for simpler error handling.
	subprocess.check_call(cmd)
	logger.info("RAID array created successfully.")
	except (subprocess.CalledProcessError, FileNotFoundError) as e:
	logger.error(f"Error during RAID creation: {str(e)}")
	sys.exit(1)