update the main branch for 2502 release #504
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: nvauto <[email protected]>
Signed-off-by: YanxuanLiu <[email protected]>
* add license header and check workflow Signed-off-by: YanxuanLiu <[email protected]> * correct xml comment format Signed-off-by: YanxuanLiu <[email protected]> --------- Signed-off-by: YanxuanLiu <[email protected]>
[auto-merge] branch-24.12 to branch-25.02 [skip ci] [bot]
[auto-merge] branch-24.12 to branch-25.02 [skip ci] [bot]
follow up of NVIDIA/spark-rapids-common#22 to avoid update action details for multiple `spark-rapids*` repos in the future Signed-off-by: Peixin Li <[email protected]>
…VIDIA#483) ### Support for running DL Inference notebooks on CSP environments. - Refactored Triton sections to use PyTriton, a Python API for the Triton inference server which avoids Docker. Once this PR is merged, Triton sections no longer need to be skipped in the CI pipeline @YanxuanLiu . - Updated notebooks with instructions to run on Databricks/Dataproc - Updated Torch notebooks with best practices for ahead-of-time TensorRT compilation. - Cleaned up README, removing instructions to start Jupyter with PySpark (we need a cell to attach to standalone for CI/CD anyway, so hoping to reduce confusion for user). Notebook outputs are saved from running locally, but all notebooks were tested on Databricks/Dataproc. --------- Signed-off-by: Rishi Chandra <[email protected]>
Pin torch<=2.5.1 until torch-tensorrt has a compatible version for torch 2.6. Add troubleshooting to update libstdc++ for Triton server binary requirements; can occur if default conda channel is used and not up to date. Removed ipykernel from requirements since it's already pulled in by jupyterlab (and slows pip down a lot). --------- Signed-off-by: Rishi Chandra <[email protected]>
Signed-off-by: Rishi Chandra <[email protected]>
Consolidate util functions into a server manager class to simplify usage. (Note that notebooks were rerun but only the Triton utility invocations are changed). Also on Dataproc, the utils file needs to be copied driver root dir instead of the same directory as the notebooks, since sc.addPyFile on Dataproc only accepts absolute paths from root. --------- Signed-off-by: Rishi Chandra <[email protected]>
Add deepseek-r1 and gemma-7b LLM batch inference notebooks. Updated CSP instructions since these notebooks require >20GB GPU RAM (A10/L4). --------- Signed-off-by: Rishi Chandra <[email protected]>
Adding Qwen notebook, mainly to demonstrate how to leverage system prompts/chat templates for batch inference. Also serves as a non-gated and faster alternative to Gemma. --------- Signed-off-by: Rishi Chandra <[email protected]>
The PR is to update the plugin version to v25.02.1 Signed-off-by: liyuan <[email protected]> Signed-off-by: liyuan <[email protected]>
Use streaming to avoid saving entire Huggingface dataset to disk for large datasets. Updated diagram for clarity regarding client/server interaction. --------- Signed-off-by: Rishi Chandra <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
update the main branch for 2502 release.
Please create a merge commit, not squash.