Skip to content

[Bug] Request cause core dump with --tp >= 2. #4152

New issue
New issue
Open
#4167
Open
[Bug] Request cause core dump with --tp >= 2.#4152
#4167
@tuilakhanh

Description

@tuilakhanh
tuilakhanh
opened on Nov 25, 2025

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

Request cause core dump with --tp >= 2 and Qwen3 MOE model.
More info

Reproduction

Serve API Server with any Qwen3 MOE model and --tp >= 2.

Request to API Server with this curl.

curl --request POST \
        --url http://{lmdeploy}/v1/chat/completions \
        --header 'content-type: application/json' \
        --data '{
    "model": "Qwen3-4B-Instruct-2507",
    "response_format": {
      "type": "json_schema",
    },
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant. Always output ONLY valid JSON following exactly this schema: {\"story_title\": string, \"language\": string, \"target_word_count\": number, \"chapters\": [ {\"chapter_number\": number, \"chapter_title\": string, \"chapter_summary\": string, \"chapter_content\": string} ] }"
      },
      {
        "role": "user",
        "content": "Hãy viết 1 câu chuyện tiếng Việt, mục tiêu khoảng 5000 từ, chia thành nhiều chương. Trả về đúng theo schema JSON mà system message đã mô tả, không được trả thêm text nào bên ngoài JSON."
      }
    ]
  }'

lmdeploy give segmentation fault (core dumped).

Some time it give stranger response like this:

{"id":"2","object":"chat.completion","created":1764067593,"model":"Qwen3-4B-Instruct-2507","choices":[{"index":0,"message":{"role":"assistant","content":"{\": {\": \"}}","gen_tokens":null,"reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":130,"total_tokens":136,"completion_tokens":6}}

Or this:

{"id":"1","object":"chat.completion","created":1764066445,"model":"Qwen3-4B-Instruct-2507","choices":[{"index":0,"message":{"role":"assistant","content":"{\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\": {\"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!","gen_tokens":null,"reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":130,"total_tokens":10000,"completion_tokens":9870}}

I also test "type": "json_schema", but still crash.

Environment

Python: 3.10.12 (main, Aug 15 2025, 14:32:43) [GCC 11.4.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1: Tesla V100-SXM2-32GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
PyTorch: 2.8.0+cu128
PyTorch compiling details: PyTorch built with:
  - GCC 13.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.8
  - NVCC architecture flags: -gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_100,code=sm_100;-gencode;arch=compute_120,code=sm_120
  - CuDNN 91.0.2  (built against CUDA 12.9)
    - Built with CuDNN 90.8
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=a1cb3cc05d46d198467bebbb6e8fba50a325d4e7, CUDA_VERSION=12.8, CUDNN_VERSION=9.8.0, CXX_COMPILER=/opt/rh/gcc-toolset-13/root/usr/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF,

TorchVision: 0.23.0+cu128
LMDeploy: 0.10.2+
transformers: 4.57.1
fastapi: 0.121.1
pydantic: 2.12.4
triton: 3.4.0
NVIDIA Topology:
	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV1	NV2	NV1	SYS	SYS	SYS	NV2	0-19,40-59	0		N/A
GPU1	NV1	 X 	NV1	NV2	SYS	SYS	NV2	SYS	0-19,40-59	0		N/A
GPU2	NV2	NV1	 X 	NV2	SYS	NV1	SYS	SYS	0-19,40-59	0		N/A
GPU3	NV1	NV2	NV2	 X 	NV1	SYS	SYS	SYS	0-19,40-59	0		N/A
GPU4	SYS	SYS	SYS	NV1	 X 	NV2	NV2	NV1	20-39,60-79	1		N/A
GPU5	SYS	SYS	NV1	SYS	NV2	 X 	NV1	NV2	20-39,60-79	1		N/A
GPU6	SYS	NV2	SYS	SYS	NV2	NV1	 X 	NV1	20-39,60-79	1		N/A
GPU7	NV2	SYS	SYS	SYS	NV1	NV2	NV1	 X 	20-39,60-79	1		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Error traceback

HINT:    Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
INFO:     Started server process [28]
INFO:     Waiting for application startup.
2025-11-25 17:47:28,017 - lmdeploy - INFO - metrics_processor.py:76 - Metrics are enabled.
2025-11-25 17:47:28,017 - lmdeploy - INFO - metrics_processor.py:103 - Metrics handler task started.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
2025-11-25 17:47:43,129 - lmdeploy - INFO - logger.py:45 - session=1, adapter_name=None, input_tokens=130, gen_config=GenerationConfig(n=1, max_new_tokens=None, do_sample=True, top_p=1.0, top_k=40, min_p=0.0, temperature=0.7, repetition_penalty=1.0, ignore_eos=False, random_seed=12475723022191024077, stop_words=None, bad_words=None, stop_token_ids=[151643, 151645], bad_token_ids=None, min_new_tokens=None, skip_special_tokens=True, spaces_between_special_tokens=True, logprobs=None, response_format={'type': 'json_object', 'json_schema': None, 'regex_schema': None}, logits_processors=None, output_logits=None, output_last_hidden_state=None, include_stop_str_in_output=False, with_cache=False, preserve_cache=False, migration_request=None), prompt='<|im_start|>system\nYou are a helpful assistant. Always output ONLY valid JSON following exactly this schema: {"story_title": string, "language": string, "target_word_count": number, "chapters": [ {"chapter_number": number, "chapter_title": string, "chapter_summary": string, "chapter_content": string} ] }<|im_end|>\n<|im_start|>user\nHãy viết 1 câu chuyện tiếng Việt, mục tiêu khoảng 5000 từ, chia thành nhiều chương. Trả về đúng theo schema JSON mà system message đã mô tả, không được trả thêm text nào bên ngoài JSON.<|im_end|>\n<|im_start|>assistant\n', prompt_token_id=[151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 23240, 2550, 26687, 2697, 4718, 2701, 6896, 419, 10802, 25, 5212, 26485, 6112, 788, 914, 11, 330, 11528, 788, 914, 11, 330, 5657, 13533, 3180, 788, 1372, 11, 330, 331, 17425, 788, 508, 5212, 40226, 5500, 788, 1372, 11, 330, 40226, 6112, 788, 914, 11, 330, 40226, 27251, 788, 914, 11, 330, 40226, 7495, 788, 914, 92, 2279, 335, 151645, 198, 151644, 872, 198, 39, 3202, 88, 128839, 220, 16, 129260, 128963, 128607, 128324, 11, 129203, 128575, 129168, 220, 20, 15, 15, 15, 94576, 11, 521, 685, 64061, 128271, 129784, 13, 1163, 21742, 128265, 129315, 279, 78, 10802, 4718, 126199, 1849, 1943, 79126, 130179, 134358, 11, 53037, 63478, 129477, 128622, 1467, 128352, 128608, 128703, 4718, 13, 151645, 198, 151644, 77091, 198]
2025-11-25 17:47:43,129 - lmdeploy - INFO - async_engine.py:816 - session=1, history_tokens=0, input_tokens=130, max_new_tokens=None, seq_start=True, seq_end=True, step=0, prep=True
2025-11-25 17:47:43,130 - lmdeploy - INFO - turbomind.py:708 - [async_stream_infer] session 1 start
[TM][INFO] Set grammar for model_request
[TM][DEBUG] Set logger level by DEBUG
[TM][INFO] [ProcessInferRequests] Request for 1 received.
[TM][INFO] [SeqMgr][Create] ID 1
[TM][WARNING] [ProcessInferRequests] [1] total sequence length (130 + 9870) exceeds `session_len` (10000), `max_new_tokens` is truncated to 9869
[TM][INFO] [Forward] [0, 1), dc=0, pf=1, sum_q=130, sum_k=130, max_q=130, max_k=130
[TM][DEBUG] void turbomind::LlamaV2::Forward(turbomind::core::Buffer_<int>, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer_<int>, turbomind::core::Buffer_<int>, turbomind::core::Buffer, turbomind::MropeRope*, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int, const turbomind::Sequence**)
[TM][DEBUG] void turbomind::LlamaV2::updateEmbedding(char*, int, const int*, const turbomind::Sequence**, int, int*, bool*)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] Set logger level by DEBUG
[TM][DEBUG] void turbomind::LlamaV2::Forward(turbomind::core::Buffer_<int>, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer_<int>, turbomind::core::Buffer_<int>, turbomind::core::Buffer, turbomind::MropeRope*, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int, const turbomind::Sequence**)
[TM][DEBUG] void turbomind::LlamaV2::updateEmbedding(char*, int, const int*, const turbomind::Sequence**, int, int*, bool*)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] turbomind::core::Tensor turbomind::LlamaV2::postDecodeEmbedding(const turbomind::core::Tensor&, turbomind::core::Buffer)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] turbomind::core::Tensor turbomind::LlamaV2::postDecodeEmbedding(const turbomind::core::Tensor&, turbomind::core::Buffer)
[TM][DEBUG] void turbomind::DynamicDecodeLayer::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&)
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::DynamicDecodeLayer::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&)
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeMaskLayer<T>::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeUpdateLayer<T>::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::GuidedDecodeMaskLayer<T>::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::GuidedDecodeUpdateLayer<T>::Setup(const std::vector<const turbomind::Request*>&, const turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::LlamaV2::dynamicDecode(turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int)
[TM][DEBUG] void turbomind::LlamaV2::dynamicDecode(turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int)
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeMaskLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeMaskLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeUpdateLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeUpdateLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::invokeStopWordsCriterion(const int*, const int*, const int*, bool*, size_t, size_t, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::invokeStopWordsCriterion(const int*, const int*, const int*, bool*, size_t, size_t, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::invokeLengthCriterion(bool*, const int*, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::invokeLengthCriterion(bool*, const int*, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][INFO] ------------------------- step = 130 -------------------------
[TM][DEBUG] void turbomind::LlamaV2::Forward(turbomind::core::Buffer_<int>, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer_<int>, turbomind::core::Buffer_<int>, turbomind::core::Buffer, turbomind::MropeRope*, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int, const turbomind::Sequence**)
[TM][DEBUG] void turbomind::LlamaV2::Forward(turbomind::core::Buffer_<int>, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer_<int>, turbomind::core::Buffer_<int>, turbomind::core::Buffer, turbomind::MropeRope*, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int, const turbomind::Sequence**)
[TM][DEBUG] void turbomind::LlamaV2::updateEmbedding(char*, int, const int*, const turbomind::Sequence**, int, int*, bool*)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::LlamaV2::updateEmbedding(char*, int, const int*, const turbomind::Sequence**, int, int*, bool*)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] turbomind::core::Tensor turbomind::LlamaV2::postDecodeEmbedding(const turbomind::core::Tensor&, turbomind::core::Buffer)
[TM][DEBUG] void turbomind::LlamaV2::dynamicDecode(turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int)
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeMaskLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] turbomind::core::Tensor turbomind::LlamaV2::postDecodeEmbedding(const turbomind::core::Tensor&, turbomind::core::Buffer)
[TM][DEBUG] void turbomind::LlamaV2::dynamicDecode(turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int)
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeMaskLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeUpdateLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeUpdateLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::invokeStopWordsCriterion(const int*, const int*, const int*, bool*, size_t, size_t, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::invokeStopWordsCriterion(const int*, const int*, const int*, bool*, size_t, size_t, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::invokeLengthCriterion(bool*, const int*, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::invokeLengthCriterion(bool*, const int*, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::LlamaV2::Forward(turbomind::core::Buffer_<int>, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer_<int>, turbomind::core::Buffer_<int>, turbomind::core::Buffer, turbomind::MropeRope*, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int, const turbomind::Sequence**)
[TM][DEBUG] void turbomind::LlamaV2::Forward(turbomind::core::Buffer_<int>, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer_<int>, turbomind::core::Buffer_<int>, turbomind::core::Buffer, turbomind::MropeRope*, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int, const turbomind::Sequence**)
[TM][DEBUG] void turbomind::LlamaV2::updateEmbedding(char*, int, const int*, const turbomind::Sequence**, int, int*, bool*)
[TM][DEBUG] void turbomind::LlamaV2::updateEmbedding(char*, int, const int*, const turbomind::Sequence**, int, int*, bool*)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] turbomind::core::Tensor turbomind::LlamaV2::postDecodeEmbedding(const turbomind::core::Tensor&, turbomind::core::Buffer)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::LlamaV2::dynamicDecode(turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int)
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeMaskLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] void turbomind::UnifiedAttentionLayer::Forward(turbomind::UnifiedAttentionLayer::ForwardParam)
[TM][DEBUG] turbomind::core::Tensor turbomind::LlamaV2::postDecodeEmbedding(const turbomind::core::Tensor&, turbomind::core::Buffer)
[TM][DEBUG] void turbomind::LlamaV2::dynamicDecode(turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Tensor, turbomind::core::Tensor, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, turbomind::core::Buffer, int, int)
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::LogitsProcessorLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeMaskLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::GuidedDecodeUpdateLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::SamplingLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
[TM][DEBUG] void turbomind::GuidedDecodeUpdateLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] start
[TM][DEBUG] void turbomind::invokeStopWordsCriterion(const int*, const int*, const int*, bool*, size_t, size_t, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::invokeLengthCriterion(bool*, const int*, int, int, int, cudaStream_t) start
[TM][DEBUG] void turbomind::StopCriteriaLayer<T>::Forward(turbomind::core::TensorMap&) [with T = float] stop
Segmentation fault (core dumped)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions