Skip to content

Add tests for custom operator implementation correctness #222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 15 commits into from
Closed

Add tests for custom operator implementation correctness #222

wants to merge 15 commits into from

Conversation

@Kacper-Pietkun
Copy link
Contributor

@Kacper-Pietkun Kacper-Pietkun commented Sep 22, 2025 •
edited
Loading

I added tests for custom ops defined in vllm_gaudi/ops:

  • For the tests of ops that are not using cuda kernels - native ops and hpu ops are triggered for the same input and their outputs are compared
  • For others tests that are using cuda kernels (so cannot be called with vllm-gaudi plugin) I created separate directory to store some predefined small tensors - weights, inputs and outputs. These tensors are too big to hardcode them in tests, however their sizes were adjusted, so all of them weight less than 3MB in total. Such tests run hpu ops with loaded inputs and weights and compare their outputs with the loaded outputs.

@Kacper-Pietkun
Add tests for some ops
9be02c8 
Signed-off-by: Kacper Pietkun <[email protected]>
@Kacper-Pietkun
add more tests (fp8)
961e87a 
Signed-off-by: Kacper Pietkun <[email protected]>
@Kacper-Pietkun Kacper-Pietkun force-pushed the dev/kpietkun/tests_custom_op_correctness branch from 3786e06 to 961e87a Compare October 3, 2025 09:37
@github-actions
Copy link

github-actions bot commented Oct 3, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@Kacper-Pietkun
remove unnecessary files
d4f1c7f 
Signed-off-by: Kacper Pietkun <[email protected]>
@github-actions
Copy link

github-actions bot commented Oct 3, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

1 similar comment
@github-actions
Copy link

github-actions bot commented Oct 3, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@Kacper-Pietkun
fix tests
ef42dd8 
Signed-off-by: Kacper Pietkun <[email protected]>
@Kacper-Pietkun Kacper-Pietkun force-pushed the dev/kpietkun/tests_custom_op_correctness branch from 500c41d to ef42dd8 Compare October 3, 2025 12:07
@github-actions
Copy link

github-actions bot commented Oct 3, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@Kacper-Pietkun
compressed_tensors tests
b1ab0e4 
Signed-off-by: Kacper Pietkun <[email protected]>
@github-actions
Copy link

github-actions bot commented Oct 8, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@github-actions
Copy link

github-actions bot commented Oct 8, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@Kacper-Pietkun
fix fp8 test
e248377 
Signed-off-by: Kacper Pietkun <[email protected]>
@github-actions
Copy link

github-actions bot commented Oct 8, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

1 similar comment
@github-actions
Copy link

github-actions bot commented Oct 9, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@Kacper-Pietkun
fused_moe test
c04da96 
Signed-off-by: Kacper Pietkun <[email protected]>
@Kacper-Pietkun Kacper-Pietkun force-pushed the dev/kpietkun/tests_custom_op_correctness branch from bd9a796 to c04da96 Compare October 9, 2025 12:02
@github-actions
Copy link

github-actions bot commented Oct 9, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@Kacper-Pietkun
multihead_attn test
dd81074 
Signed-off-by: Kacper Pietkun <[email protected]>
@github-actions
Copy link

github-actions bot commented Oct 16, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@github-actions
Copy link

github-actions bot commented Oct 16, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@Kacper-Pietkun
clean code
2aeafe2 
Signed-off-by: Kacper Pietkun <[email protected]>
@github-actions
Copy link

github-actions bot commented Oct 16, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@Kacper-Pietkun
fix utils
79b5af8 
Signed-off-by: Kacper Pietkun <[email protected]>
@github-actions
Copy link

github-actions bot commented Oct 16, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

This is a Draft PR. Please mark it as 'Ready for Review' to trigger the CI.

@Kacper-Pietkun Kacper-Pietkun marked this pull request as ready for review October 16, 2025 07:43
@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 16, 2025

/run-gaudi-tests

@Kacper-Pietkun
fix mypy
b657874 
Signed-off-by: Kacper Pietkun <[email protected]>
@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 16, 2025

/run-gaudi-tests

1 similar comment
@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 20, 2025

/run-gaudi-tests

@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 21, 2025

/run-gaudi-tests

@Kacper-Pietkun Kacper-Pietkun force-pushed the dev/kpietkun/tests_custom_op_correctness branch from 863ceec to 8a18ad6 Compare October 21, 2025 09:51
@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 21, 2025

/run-gaudi-tests

1 similar comment
@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 21, 2025

/run-gaudi-tests

@Kacper-Pietkun Kacper-Pietkun force-pushed the dev/kpietkun/tests_custom_op_correctness branch from 8c5485b to 2245bae Compare October 21, 2025 09:57
@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 21, 2025

/run-gaudi-tests

4 similar comments
@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 21, 2025

/run-gaudi-tests

@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 21, 2025

/run-gaudi-tests

@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 22, 2025

/run-gaudi-tests

@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 22, 2025

/run-gaudi-tests

@Kacper-Pietkun
adjust fp8 tests for gaudi2
e9e95fa 
Signed-off-by: Kacper Pietkun <[email protected]>
@Kacper-Pietkun Kacper-Pietkun force-pushed the dev/kpietkun/tests_custom_op_correctness branch from f4d79d3 to e9e95fa Compare October 23, 2025 11:53
@github-actions
Copy link

github-actions bot commented Oct 23, 2025

🚧 CI Blocked

The main CI workflow was not started for the following reason:

Your branch is behind the base branch. Please merge or rebase to get the latest changes.

@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 23, 2025

/run-gaudi-tests

@Kacper-Pietkun Kacper-Pietkun force-pushed the dev/kpietkun/tests_custom_op_correctness branch from 9da965e to b9777f8 Compare October 23, 2025 12:13
@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 23, 2025

/run-gaudi-tests

@Kacper-Pietkun Kacper-Pietkun marked this pull request as draft October 23, 2025 12:52
@Kacper-Pietkun
Copy link
Contributor Author

Kacper-Pietkun commented Oct 23, 2025 •
edited
Loading

I am closing this draft PR and I will create a new clean PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kzawora-intel kzawora-intel Awaiting requested review from kzawora-intel kzawora-intel is a code owner

@xuechendi xuechendi Awaiting requested review from xuechendi xuechendi is a code owner

@mswiniarsk mswiniarsk Awaiting requested review from mswiniarsk mswiniarsk is a code owner

@adobrzyn adobrzyn Awaiting requested review from adobrzyn adobrzyn is a code owner

@mgawarkiewicz-intel mgawarkiewicz-intel Awaiting requested review from mgawarkiewicz-intel mgawarkiewicz-intel is a code owner

@vivekgoe vivekgoe Awaiting requested review from vivekgoe vivekgoe is a code owner

@afierka-intel afierka-intel Awaiting requested review from afierka-intel afierka-intel is a code owner

@michalkuligowski michalkuligowski Awaiting requested review from michalkuligowski michalkuligowski is a code owner

@iboiko-habana iboiko-habana Awaiting requested review from iboiko-habana iboiko-habana is a code owner

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Kacper-Pietkun