Repository navigation
How to pass parameter in ensemble model? #71
Description
As the normal procedure for tensorrtllm_backend is preprocessing -> (tensorrt_llm) process -> postprocessing. How to pass the customer parameter from the request, like request token length.
In my understanding, tensorrt_llm backend will finish the infer, it won't work to add input and output parameter. then the issue coming, in ensemble pipeline, how to pass the parameter from the preprocess module to poseprocess module.
Please any way to solve this issue?
Metadata
Metadata
Assignees
Labels
No labels
Activity
byshiue commentedon Nov 1, 2023
I think I don't really get your question.
max_tokens(here) is a input form parameter, which is similar torequest token length. Can you explain more for your question?GooVincent commentedon Nov 2, 2023
Well, I understood how to pass the parameter from the model preprocess to model tensorrt_llm, like REQUEST_INPUT_LEN
but how to pass REQUEST_INPUT_LEN from model tensorrt_llm to the model postprocess, I get confused, as the output for tensorrt_llm should be fixed based the llm model. correct?
byshiue commentedon Nov 2, 2023
For ensemble, you should pass
REQUEST_INPUT_LENforensembletopostprocess, instead of passing fromtensorrt_llmtopostprocess.GooVincent commentedon Nov 2, 2023
Any guide for this please? as the REQUEST_INPUT_LEN is the intermediate result.
byshiue commentedon Nov 2, 2023
I get your point. As far as I know, there is no way to support such feature in tensorrt_llm backend directly because it needs to change the source code of batch_manager. You can try to map the output of preporcess to input of postprocess directly. I am not sure is it doable. You can ask in tritonserver repo.
GooVincent commentedon Nov 2, 2023
okay, thanks.
GooVincent commentedon Nov 2, 2023
Any way for the ensemble mode to know the request token length, as I wanna cut the original tokens.
BasicCoder commentedon Nov 3, 2023
May be you can reference this #95
callmezhangchenchenokay commentedon Nov 8, 2023
@GooVincent Hi ，
Have you solved it yet
callmezhangchenchenokay commentedon Dec 8, 2023
Here is a demo, you can pass out the REQUEST_INPUT_LEN in the middle,
I've tested it and it's okay
https://github.com/triton-inference-server/tensorrtllm_backend/blob/57de9f572f75f61fe17b668eea1430b030e1b721/all_models/inflight_batcher_llm/postprocessing/config.pbtxt
byshiue commentedon Dec 11, 2023
Since the issue is solved, close this bug. If you still have issue/question, feel free to ask and we will open the issue again.
callmezhangchenchenokay commentedon Dec 29, 2023
Sorry to interrupt again!
The solution mentioned above requires model_transaction_policy to be set to True,
Can only be used if stream = False,
However, this problem occurs when stream = True
So there needs to be a way to export REQUEST_INPUT_LEN from stream =True, model_transaction_policy =True
byshiue commentedon Jan 3, 2024
I don't get you point of "set model_transaction_policy to be True".
model_transaction_policyis a set of policy and stream (decoupled mode) is one of them.callmezhangchenchenokay commentedon Jan 4, 2024
Excuse me! It's my expression
What I'm trying to say is,
‘’‘
model_transaction_policy {
decoupled: true
}
’‘’
(decoupled) If set to False, can export REQUEST_INPUT_LEN,
However, when decoupled is set to True, the following error is reported
byshiue commentedon Jan 8, 2024
Could you try scripts here https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llama.md#end-to-end-workflow-to-run-llama to run the streamming?
Add GitHub action to format and lint code (#71)