Why is sglang significantly lower than vllm in my benchmark?

[HelloCard]

Operating system: win10 x64, wsl2
Hardware: 2080ti-22G*2, nvlink (SLi)
Test script: https://github.com/vllm-project/vllm/tree/main/benchmarks
Test command: python benchmark_serving.py --model /mnt/e/Code/models/microsoft-Orca-2-13b-8-bit-gptq --dataset-path /mnt/e/Code/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 5000 --request-rate 3.7(may change) --backend openai --base-url http://0.0.0.0:8000 --endpoint /v1/completions
Pip package version:
Python 3.11.8
vllm 0.5.5
vllm-flash-attn 2.6.1
flashinfer 0.1.6+cu121torch2.4
sglang 0.2.14

Roughly speaking, this test script selects num-prompts statements with a length of no more than 1024 tokens from ShareGPT_V3_unfiltered_cleaned_split.json, and asks the API to continue writing text of the same length.
I am pretty sure that vllm does not cache these requests. All tests were performed after restarting the API service.

Results:

python3 -m sglang.launch_server --model-path /mnt/e/Code/models/microsoft-Orca-2-13b-8-bit-gptq --tensor-parallel-size 2 --port 8000 --context-length 4096 --mem-fraction-static 0.80
Traffic request rate: 4.1
============ Serving Benchmark Result ============
Successful requests:                     3803
Benchmark duration (s):                  1621.03
Total input tokens:                      941007
Total generated tokens:                  579898
Request throughput (req/s):              2.35
Input token throughput (tok/s):          580.50
Output token throughput (tok/s):         357.73
---------------Time to First Token----------------
Mean TTFT (ms):                          177799.78
Median TTFT (ms):                        5680.62
P99 TTFT (ms):                           823119.72
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          221.18
Median TPOT (ms):                        205.34
P99 TPOT (ms):                           539.98
---------------Inter-token Latency----------------
Mean ITL (ms):                           1381.97
Median ITL (ms):                         142.76
P99 ITL (ms):                            1547.15
==================================================


python3 -m sglang.launch_server --model-path /mnt/e/Code/models/microsoft-Orca-2-13b-8-bit-gptq --tensor-parallel-size 2 --port 8000 --context-length 4096 --max-prefill-tokens 32768 --mem-fraction-static 0.80
Traffic request rate: 4.1
============ Serving Benchmark Result ============
Successful requests:                     3874
Benchmark duration (s):                  1617.57
Total input tokens:                      959844
Total generated tokens:                  579552
Request throughput (req/s):              2.39
Input token throughput (tok/s):          593.38
Output token throughput (tok/s):         358.28
---------------Time to First Token----------------
Mean TTFT (ms):                          169079.65
Median TTFT (ms):                        4716.93
P99 TTFT (ms):                           813122.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          225.48
Median TPOT (ms):                        219.43
P99 TPOT (ms):                           520.09
---------------Inter-token Latency----------------
Mean ITL (ms):                           1348.66
Median ITL (ms):                         145.17
P99 ITL (ms):                            1633.98
==================================================


python3 -m vllm.entrypoints.openai.api_server --model /mnt/e/Code/models/microsoft-Orca-2-13b-8-bit-gptq --trust-remote-code --max-model-len 4096 --tensor-parallel-size 2 --gpu-memory-utilization 0.80
Traffic request rate: 4.0
============ Serving Benchmark Result ============
Successful requests:                     5000
Benchmark duration (s):                  1432.18
Total input tokens:                      1235887
Total generated tokens:                  758318
Request throughput (req/s):              3.49
Input token throughput (tok/s):          862.94
Output token throughput (tok/s):         529.48
---------------Time to First Token----------------
Mean TTFT (ms):                          24452.21
Median TTFT (ms):                        14329.36
P99 TTFT (ms):                           100574.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          176.11
Median TPOT (ms):                        168.90
P99 TPOT (ms):                           336.97
---------------Inter-token Latency----------------
Mean ITL (ms):                           329.22
Median ITL (ms):                         120.26
P99 ITL (ms):                            1009.21
==================================================

As shown in the above three tests, the request processing speed of sglang does not exceed 2.5request/s, while vllm is around 3.5. There is also a big gap in output token throughput. I adjusted the max-prefill-tokens parameter and re-ran sglang, but the situation did not change significantly.
How do I need to tune sglang's parameters to improve performance? Or is this phenomenon caused by the poor performance of flashinfer on 2080ti?

[HelloCard]

python3 -m sglang.launch_server --model-path /mnt/e/Code/models/microsoft-Orca-2-13b-8-bit-gptq --tensor-parallel-size 2 --port 8000 --context-length 4096 --max-prefill-tokens 32768 --mem-fraction-static 0.80
[19:21:54 TP0] Decode batch. #running-req: 87, #token: 21425, token usage: 0.82, gen throughput (token/s): 170.51, #queue-req: 3785  
[19:22:08 TP0] Decode batch. #running-req: 78, #token: 22419, token usage: 0.86, gen throughput (token/s): 229.82, #queue-req: 3787  
[19:22:22 TP0] Decode batch. #running-req: 72, #token: 23822, token usage: 0.91, gen throughput (token/s): 220.00, #queue-req: 3787  
[19:22:39 TP0] Decode batch. #running-req: 84, #token: 23821, token usage: 0.91, gen throughput (token/s): 164.01, #queue-req: 3766  
[19:22:57 TP0] Decode batch. #running-req: 92, #token: 23422, token usage: 0.90, gen throughput (token/s): 191.63, #queue-req: 3744  
[19:23:11 TP0] Decode batch. #running-req: 85, #token: 24037, token usage: 0.92, gen throughput (token/s): 245.40, #queue-req: 3741  
[19:23:25 TP0] Decode batch. #running-req: 79, #token: 26015, token usage: 0.99, gen throughput (token/s): 244.49, #queue-req: 3741  
[19:23:39 TP0] Decode batch. #running-req: 67, #token: 24949, token usage: 0.95, gen throughput (token/s): 204.40, #queue-req: 3744  
[19:23:52 TP0] Decode batch. #running-req: 60, #token: 23687, token usage: 0.91, gen throughput (token/s): 192.43, #queue-req: 3744  
[19:24:06 TP0] Decode batch. #running-req: 50, #token: 21215, token usage: 0.81, gen throughput (token/s): 164.40, #queue-req: 3744  
[19:24:23 TP0] Decode batch. #running-req: 56, #token: 17440, token usage: 0.67, gen throughput (token/s): 118.97, #queue-req: 3726  
[19:24:40 TP0] Decode batch. #running-req: 63, #token: 17377, token usage: 0.66, gen throughput (token/s): 145.60, #queue-req: 3712  
[19:24:56 TP0] Decode batch. #running-req: 69, #token: 18588, token usage: 0.71, gen throughput (token/s): 172.38, #queue-req: 3702  
[19:25:10 TP0] Decode batch. #running-req: 67, #token: 20829, token usage: 0.80, gen throughput (token/s): 196.04, #queue-req: 3702  
[19:25:25 TP0] Decode batch. #running-req: 64, #token: 21383, token usage: 0.82, gen throughput (token/s): 174.50, #queue-req: 3700  
[19:25:45 TP0] Decode batch. #running-req: 75, #token: 21140, token usage: 0.81, gen throughput (token/s): 133.37, #queue-req: 3678  
[19:26:02 TP0] Decode batch. #running-req: 71, #token: 23301, token usage: 0.89, gen throughput (token/s): 182.31, #queue-req: 3673  
[19:26:20 TP0] Decode batch. #running-req: 87, #token: 23288, token usage: 0.89, gen throughput (token/s): 184.66, #queue-req: 3648  
[19:26:37 TP0] Decode batch. #running-req: 79, #token: 22655, token usage: 0.87, gen throughput (token/s): 208.52, #queue-req: 3642  
[19:26:53 TP0] Decode batch. #running-req: 80, #token: 24046, token usage: 0.92, gen throughput (token/s): 202.42, #queue-req: 3631  
[19:27:11 TP0] Decode batch. #running-req: 87, #token: 23577, token usage: 0.90, gen throughput (token/s): 179.74, #queue-req: 3614  
[19:27:25 TP0] Decode batch. #running-req: 79, #token: 23335, token usage: 0.89, gen throughput (token/s): 233.38, #queue-req: 3611  
[19:27:38 TP0] Decode batch. #running-req: 73, #token: 23980, token usage: 0.92, gen throughput (token/s): 226.86, #queue-req: 3611  
[19:27:51 TP0] Decode batch. #running-req: 65, #token: 24589, token usage: 0.94, gen throughput (token/s): 206.82, #queue-req: 3611  
[19:28:05 TP0] Decode batch. #running-req: 59, #token: 25183, token usage: 0.96, gen throughput (token/s): 179.66, #queue-req: 3611  
[19:28:27 TP0] Decode batch. #running-req: 80, #token: 24248, token usage: 0.93, gen throughput (token/s): 134.12, #queue-req: 3578  
[19:28:42 TP0] Decode batch. #running-req: 81, #token: 24301, token usage: 0.93, gen throughput (token/s): 221.64, #queue-req: 3569  
[19:28:56 TP0] Decode batch. #running-req: 78, #token: 23868, token usage: 0.91, gen throughput (token/s): 218.47, #queue-req: 3563  
[19:29:09 TP0] Decode batch. #running-req: 71, #token: 24195, token usage: 0.92, gen throughput (token/s): 235.03, #queue-req: 3563  
[19:29:28 TP0] Decode batch. #running-req: 88, #token: 23509, token usage: 0.90, gen throughput (token/s): 165.99, #queue-req: 3537  
[19:29:45 TP0] Decode batch. #running-req: 89, #token: 24589, token usage: 0.94, gen throughput (token/s): 223.16, #queue-req: 3529  
[19:29:59 TP0] Decode batch. #running-req: 80, #token: 25002, token usage: 0.96, gen throughput (token/s): 240.41, #queue-req: 3529  
[19:30:12 TP0] Decode batch. #running-req: 74, #token: 25434, token usage: 0.97, gen throughput (token/s): 222.60, #queue-req: 3529  
[19:30:28 TP0] Decode batch. #running-req: 67, #token: 22659, token usage: 0.87, gen throughput (token/s): 181.64, #queue-req: 3522  
[19:30:43 TP0] Decode batch. #running-req: 66, #token: 24362, token usage: 0.93, gen throughput (token/s): 186.28, #queue-req: 3510  
[19:30:56 TP0] Decode batch. #running-req: 60, #token: 24909, token usage: 0.95, gen throughput (token/s): 186.41, #queue-req: 3509  
[19:31:13 TP0] Decode batch. #running-req: 63, #token: 24816, token usage: 0.95, gen throughput (token/s): 139.16, #queue-req: 3494  
[19:31:30 TP0] Decode batch. #running-req: 59, #token: 24541, token usage: 0.94, gen throughput (token/s): 142.32, #queue-req: 3482  
[19:31:49 TP0] Decode batch. #running-req: 55, #token: 24506, token usage: 0.94, gen throughput (token/s): 128.65, #queue-req: 3464  
[19:32:07 TP0] Decode batch. #running-req: 51, #token: 24817, token usage: 0.95, gen throughput (token/s): 118.34, #queue-req: 3449  
[19:32:25 TP0] Decode batch. #running-req: 51, #token: 24970, token usage: 0.95, gen throughput (token/s): 109.79, #queue-req: 3435  
[19:32:43 TP0] Decode batch. #running-req: 53, #token: 24912, token usage: 0.95, gen throughput (token/s): 109.73, #queue-req: 3415  
[19:33:02 TP0] Decode batch. #running-req: 51, #token: 25051, token usage: 0.96, gen throughput (token/s): 113.03, #queue-req: 3398  
[19:33:17 TP0] Decode batch. #running-req: 50, #token: 24945, token usage: 0.95, gen throughput (token/s): 124.22, #queue-req: 3388  
[19:33:35 TP0] Decode batch. #running-req: 42, #token: 23354, token usage: 0.89, gen throughput (token/s): 105.61, #queue-req: 3380  
[19:33:58 TP0] Decode batch. #running-req: 56, #token: 24085, token usage: 0.92, gen throughput (token/s): 80.04, #queue-req: 3349   
[19:34:22 TP0] Decode batch. #running-req: 71, #token: 23172, token usage: 0.89, gen throughput (token/s): 112.29, #queue-req: 3319  
[19:34:38 TP0] Decode batch. #running-req: 69, #token: 24564, token usage: 0.94, gen throughput (token/s): 173.64, #queue-req: 3316  
[19:34:51 TP0] Decode batch. #running-req: 59, #token: 24627, token usage: 0.94, gen throughput (token/s): 204.06, #queue-req: 3320  
[19:35:04 TP0] Decode batch. #running-req: 56, #token: 25410, token usage: 0.97, gen throughput (token/s): 176.48, #queue-req: 3320  
[19:35:17 TP0] Decode batch. #running-req: 48, #token: 24997, token usage: 0.96, gen throughput (token/s): 164.51, #queue-req: 3324  
[19:35:30 TP0] Decode batch. #running-req: 41, #token: 22524, token usage: 0.86, gen throughput (token/s): 129.87, #queue-req: 3324  
[19:35:44 TP0] Decode batch. #running-req: 37, #token: 20595, token usage: 0.79, gen throughput (token/s): 115.94, #queue-req: 3323  
[19:36:01 TP0] Decode batch. #running-req: 44, #token: 19865, token usage: 0.76, gen throughput (token/s): 100.29, #queue-req: 3310  
[19:36:15 TP0] Decode batch. #running-req: 45, #token: 20370, token usage: 0.78, gen throughput (token/s): 129.45, #queue-req: 3304  
[19:36:29 TP0] Decode batch. #running-req: 43, #token: 21036, token usage: 0.80, gen throughput (token/s): 124.76, #queue-req: 3303  
[19:36:42 TP0] Decode batch. #running-req: 42, #token: 22795, token usage: 0.87, gen throughput (token/s): 126.86, #queue-req: 3300  
[19:36:56 TP0] Decode batch. #running-req: 40, #token: 22254, token usage: 0.85, gen throughput (token/s): 115.18, #queue-req: 3295  
[19:37:12 TP0] Decode batch. #running-req: 41, #token: 23128, token usage: 0.88, gen throughput (token/s): 99.39, #queue-req: 3287   
[19:37:31 TP0] Decode batch. #running-req: 41, #token: 23179, token usage: 0.89, gen throughput (token/s): 86.74, #queue-req: 3273   
[19:37:47 TP0] Decode batch. #running-req: 41, #token: 23814, token usage: 0.91, gen throughput (token/s): 104.92, #queue-req: 3262  
[19:38:06 TP0] Decode batch. #running-req: 46, #token: 24321, token usage: 0.93, gen throughput (token/s): 95.26, #queue-req: 3241   
[19:38:22 TP0] Decode batch. #running-req: 49, #token: 24467, token usage: 0.94, gen throughput (token/s): 114.95, #queue-req: 3221  
[19:38:39 TP0] Decode batch. #running-req: 49, #token: 24613, token usage: 0.94, gen throughput (token/s): 117.10, #queue-req: 3208  
[19:38:57 TP0] Decode batch. #running-req: 53, #token: 24725, token usage: 0.94, gen throughput (token/s): 117.52, #queue-req: 3190  
[19:39:10 TP0] Decode batch. #running-req: 49, #token: 24779, token usage: 0.95, gen throughput (token/s): 148.83, #queue-req: 3184  
[19:39:25 TP0] Decode batch. #running-req: 48, #token: 25052, token usage: 0.96, gen throughput (token/s): 131.15, #queue-req: 3174  
[19:39:42 TP0] Decode batch. #running-req: 47, #token: 24150, token usage: 0.92, gen throughput (token/s): 107.83, #queue-req: 3159  
[19:39:58 TP0] Decode batch. #running-req: 47, #token: 24136, token usage: 0.92, gen throughput (token/s): 117.61, #queue-req: 3147  
[19:40:10 TP0] Decode batch. #running-req: 47, #token: 25029, token usage: 0.96, gen throughput (token/s): 151.19, #queue-req: 3144  
[19:40:24 TP0] Decode batch. #running-req: 45, #token: 24286, token usage: 0.93, gen throughput (token/s): 133.85, #queue-req: 3138  
[19:40:40 TP0] Decode batch. #running-req: 48, #token: 24187, token usage: 0.92, gen throughput (token/s): 117.17, #queue-req: 3125  
[19:40:53 TP0] Decode batch. #running-req: 46, #token: 25409, token usage: 0.97, gen throughput (token/s): 142.64, #queue-req: 3125  
[19:41:06 TP0] Decode batch. #running-req: 41, #token: 24682, token usage: 0.94, gen throughput (token/s): 129.71, #queue-req: 3124  
[19:41:21 TP0] Decode batch. #running-req: 43, #token: 24521, token usage: 0.94, gen throughput (token/s): 114.50, #queue-req: 3117  
[19:41:37 TP0] Decode batch. #running-req: 45, #token: 25095, token usage: 0.96, gen throughput (token/s): 116.69, #queue-req: 3109  
[19:41:52 TP0] Decode batch. #running-req: 44, #token: 24879, token usage: 0.95, gen throughput (token/s): 121.13, #queue-req: 3104  
[19:42:11 TP0] Decode batch. #running-req: 71, #token: 24837, token usage: 0.95, gen throughput (token/s): 130.13, #queue-req: 3068  
[19:42:23 TP0] Decode batch. #running-req: 55, #token: 25091, token usage: 0.96, gen throughput (token/s): 215.76, #queue-req: 3079  
[19:42:36 TP0] Decode batch. #running-req: 45, #token: 24841, token usage: 0.95, gen throughput (token/s): 157.91, #queue-req: 3087  
[19:42:48 TP0] Decode batch. #running-req: 41, #token: 25148, token usage: 0.96, gen throughput (token/s): 146.83, #queue-req: 3090  
[19:43:01 TP0] Decode batch. #running-req: 36, #token: 24643, token usage: 0.94, gen throughput (token/s): 118.39, #queue-req: 3090  
[19:43:14 TP0] Decode batch. #running-req: 39, #token: 20616, token usage: 0.79, gen throughput (token/s): 102.13, #queue-req: 3079  
[19:43:26 TP0] Decode batch. #running-req: 37, #token: 21007, token usage: 0.80, gen throughput (token/s): 132.30, #queue-req: 3078  
[19:43:40 TP0] Decode batch. #running-req: 40, #token: 21863, token usage: 0.84, gen throughput (token/s): 109.60, #queue-req: 3072  
[19:43:58 TP0] Decode batch. #running-req: 58, #token: 20172, token usage: 0.77, gen throughput (token/s): 109.29, #queue-req: 3046  
[19:44:15 TP0] Decode batch. #running-req: 61, #token: 19361, token usage: 0.74, gen throughput (token/s): 142.96, #queue-req: 3023  
[19:44:28 TP0] Decode batch. #running-req: 60, #token: 20379, token usage: 0.78, gen throughput (token/s): 182.59, #queue-req: 3016  
[19:44:40 TP0] Decode batch. #running-req: 57, #token: 22243, token usage: 0.85, gen throughput (token/s): 198.63, #queue-req: 3014  
[19:44:53 TP0] Decode batch. #running-req: 57, #token: 22073, token usage: 0.84, gen throughput (token/s): 168.37, #queue-req: 3005  
[19:45:09 TP0] Decode batch. #running-req: 52, #token: 19503, token usage: 0.75, gen throughput (token/s): 143.97, #queue-req: 2991  
[19:45:32 TP0] Decode batch. #running-req: 90, #token: 20357, token usage: 0.78, gen throughput (token/s): 124.08, #queue-req: 2930  
[19:45:47 TP0] Decode batch. #running-req: 85, #token: 21994, token usage: 0.84, gen throughput (token/s): 239.55, #queue-req: 2925  
[19:46:00 TP0] Decode batch. #running-req: 79, #token: 23479, token usage: 0.90, gen throughput (token/s): 255.18, #queue-req: 2922  
[19:46:15 TP0] Decode batch. #running-req: 78, #token: 23905, token usage: 0.91, gen throughput (token/s): 213.23, #queue-req: 2910  
[19:46:28 TP0] Decode batch. #running-req: 72, #token: 24258, token usage: 0.93, gen throughput (token/s): 215.87, #queue-req: 2900  
[19:46:42 TP0] Decode batch. #running-req: 69, #token: 25014, token usage: 0.96, gen throughput (token/s): 208.55, #queue-req: 2894  
[19:46:54 TP0] Decode batch. #running-req: 57, #token: 23722, token usage: 0.91, gen throughput (token/s): 207.55, #queue-req: 2892  
[19:47:10 TP0] Decode batch. #running-req: 52, #token: 23858, token usage: 0.91, gen throughput (token/s): 148.76, #queue-req: 2883  
[19:47:27 TP0] Decode batch. #running-req: 50, #token: 24005, token usage: 0.92, gen throughput (token/s): 118.46, #queue-req: 2865  
[19:47:44 TP0] Decode batch. #running-req: 44, #token: 23540, token usage: 0.90, gen throughput (token/s): 113.41, #queue-req: 2852  
[19:48:08 TP0] Decode batch. #running-req: 51, #token: 25032, token usage: 0.96, gen throughput (token/s): 86.82, #queue-req: 2811   
[19:48:23 TP0] Decode batch. #running-req: 46, #token: 24748, token usage: 0.95, gen throughput (token/s): 125.04, #queue-req: 2800  
[19:48:39 TP0] Decode batch. #running-req: 46, #token: 24640, token usage: 0.94, gen throughput (token/s): 116.98, #queue-req: 2783  
[19:48:58 TP0] Decode batch. #running-req: 44, #token: 24034, token usage: 0.92, gen throughput (token/s): 94.22, #queue-req: 2756   
[19:49:16 TP0] Decode batch. #running-req: 43, #token: 24903, token usage: 0.95, gen throughput (token/s): 99.54, #queue-req: 2736   
[19:49:32 TP0] Decode batch. #running-req: 40, #token: 24154, token usage: 0.92, gen throughput (token/s): 107.40, #queue-req: 2719  
[19:49:48 TP0] Decode batch. #running-req: 41, #token: 24888, token usage: 0.95, gen throughput (token/s): 97.44, #queue-req: 2703   
[19:50:07 TP0] Decode batch. #running-req: 36, #token: 24346, token usage: 0.93, gen throughput (token/s): 81.45, #queue-req: 2687   
[19:50:24 TP0] Decode batch. #running-req: 33, #token: 24068, token usage: 0.92, gen throughput (token/s): 81.67, #queue-req: 2676   
[19:50:43 TP0] Decode batch. #running-req: 41, #token: 24788, token usage: 0.95, gen throughput (token/s): 80.64, #queue-req: 2650   
[19:51:02 TP0] Decode batch. #running-req: 49, #token: 24191, token usage: 0.92, gen throughput (token/s): 88.64, #queue-req: 2629   
[19:51:17 TP0] Decode batch. #running-req: 47, #token: 23884, token usage: 0.91, gen throughput (token/s): 131.56, #queue-req: 2622  
[19:51:31 TP0] Decode batch. #running-req: 48, #token: 24092, token usage: 0.92, gen throughput (token/s): 139.41, #queue-req: 2617  
[19:51:46 TP0] Decode batch. #running-req: 48, #token: 24313, token usage: 0.93, gen throughput (token/s): 121.38, #queue-req: 2606  
[19:52:04 TP0] Decode batch. #running-req: 51, #token: 24044, token usage: 0.92, gen throughput (token/s): 121.17, #queue-req: 2587  
[19:52:21 TP0] Decode batch. #running-req: 50, #token: 24846, token usage: 0.95, gen throughput (token/s): 114.95, #queue-req: 2573  
[19:52:36 TP0] Decode batch. #running-req: 49, #token: 23894, token usage: 0.91, gen throughput (token/s): 137.17, #queue-req: 2565  
[19:52:50 TP0] Decode batch. #running-req: 47, #token: 25527, token usage: 0.98, gen throughput (token/s): 134.49, #queue-req: 2563  
[19:53:03 TP0] Decode batch. #running-req: 40, #token: 23389, token usage: 0.89, gen throughput (token/s): 138.07, #queue-req: 2566  
[19:53:16 TP0] Decode batch. #running-req: 36, #token: 23149, token usage: 0.88, gen throughput (token/s): 113.02, #queue-req: 2566  
[19:53:30 TP0] Decode batch. #running-req: 33, #token: 22957, token usage: 0.88, gen throughput (token/s): 100.08, #queue-req: 2566  
[19:53:43 TP0] Decode batch. #running-req: 31, #token: 23342, token usage: 0.89, gen throughput (token/s): 95.02, #queue-req: 2566   
[19:53:57 TP0] Decode batch. #running-req: 31, #token: 23020, token usage: 0.88, gen throughput (token/s): 88.56, #queue-req: 2559   
[19:54:17 TP0] Decode batch. #running-req: 68, #token: 20028, token usage: 0.77, gen throughput (token/s): 104.92, #queue-req: 2510  
[19:54:32 TP0] Decode batch. #running-req: 72, #token: 19953, token usage: 0.76, gen throughput (token/s): 202.42, #queue-req: 2496  
[19:54:47 TP0] Decode batch. #running-req: 78, #token: 19888, token usage: 0.76, gen throughput (token/s): 192.26, #queue-req: 2474  
[19:55:01 TP0] Decode batch. #running-req: 71, #token: 20706, token usage: 0.79, gen throughput (token/s): 207.88, #queue-req: 2464  
[19:55:14 TP0] Decode batch. #running-req: 65, #token: 21380, token usage: 0.82, gen throughput (token/s): 211.32, #queue-req: 2459  
[19:55:25 TP0] Decode batch. #running-req: 60, #token: 22108, token usage: 0.84, gen throughput (token/s): 240.58, #queue-req: 2457  
[19:55:37 TP0] Decode batch. #running-req: 59, #token: 23805, token usage: 0.91, gen throughput (token/s): 206.98, #queue-req: 2450  
[19:55:50 TP0] Decode batch. #running-req: 56, #token: 24276, token usage: 0.93, gen throughput (token/s): 178.96, #queue-req: 2444  
[19:56:05 TP0] Decode batch. #running-req: 68, #token: 24206, token usage: 0.93, gen throughput (token/s): 155.71, #queue-req: 2417  
[19:56:17 TP0] Decode batch. #running-req: 58, #token: 23365, token usage: 0.89, gen throughput (token/s): 212.24, #queue-req: 2412  
[19:56:27 TP0] Decode batch. #running-req: 55, #token: 24426, token usage: 0.93, gen throughput (token/s): 219.62, #queue-req: 2412  
[19:56:37 TP0] Decode batch. #running-req: 52, #token: 25194, token usage: 0.96, gen throughput (token/s): 232.03, #queue-req: 2412  
[19:56:48 TP0] Decode batch. #running-req: 48, #token: 24784, token usage: 0.95, gen throughput (token/s): 162.93, #queue-req: 2408  
[19:57:00 TP0] Decode batch. #running-req: 41, #token: 24701, token usage: 0.94, gen throughput (token/s): 156.82, #queue-req: 2403  




(base) root@DESKTOP-O6DNFE1:/mnt/e/Code/Qwen7B/benchmarks# python3 -m sglang.bench_serving --backend sglang --dataset-name sharegpt --dataset-path /mnt/e/Code/Qwen7B/ShareGPT_V3_unfiltered_cleaned_split.json --model /mnt/e/Code/models/microsoft-Orca-2-13b-8-bit-gptq --num-prompts 5000 --request-rate 4.0 --base-url http://0.0.0.0:8000

WARNING It is recommended to use the `Chat` or `Instruct` model for benchmarking.
Because when the tokenizer counts the output tokens, if there is gibberish, it might count incorrectly.

Namespace(backend='sglang', base_url='http://0.0.0.0:8000', host='0.0.0.0', port=30000, dataset_name='sharegpt', dataset_path='/mnt/e/Code/Qwen7B/ShareGPT_V3_unfiltered_cleaned_split.json', model='/mnt/e/Code/models/microsoft-Orca-2-13b-8-bit-gptq', tokenizer=None, num_prompts=5000, sharegpt_output_len=None, random_input_len=1024, random_output_len=128, random_range_ratio=0.0, request_rate=4.0, seed=1, multi=False, request_rate_range='2,34,2', output_file=None, disable_tqdm=False, disable_stream=False, disable_ignore_eos=False, extra_request_body=None)

Token indices sequence length is longer than the specified maximum sequence length for this model (6048 > 4096). Running this sequence through the model will result in indexing errors
Starting initial single prompt test run...
Initial test run completed. Starting main benchmark run...
100%|█████████████████████████████████████████████████████████████████████████████| 5000/5000 [1:27:58<00:00,  1.06s/it]

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    4.0
Successful requests:                     5000
Benchmark duration (s):                  5278.60
Total input tokens:                      1282536
Total generated tokens:                  1140975
Total generated tokens (retokenized):    1136975
Request throughput (req/s):              0.95
Input token throughput (tok/s):          242.97
Output token throughput (tok/s):         216.15
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2297329.74
Median E2E Latency (ms):                 2635369.63
---------------Time to First Token----------------
Mean TTFT (ms):                          2235368.15
Median TTFT (ms):                        2580790.71
P99 TTFT (ms):                           4477034.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          284.70
Median TPOT (ms):                        276.87
P99 TPOT (ms):                           578.27
---------------Inter-token Latency----------------
Mean ITL (ms):                           275.38
Median ITL (ms):                         215.14
P99 ITL (ms):                            1134.44
==================================================

Using the sglang benchmark program, the results are even worse.

[zhyncs]

2080Ti is sm75. We haven't tested or optimized it on this, and recommend using data center level devices such as the A100, H100.

[HelloCard]

I am not sure, if I need to benchmark again, should I wait for sglang's optimization about sm75, or flashinfer's optimization about sm75?

[zhyncs]

Overall it's not a high priority for us to optimize on sm75. We suggest using the A10, A100, L40S, or H100 devices instead for now.

[HelloCard]

Okay, that's a shame.

[zhyncs]

BTW if you want to use sm70(V100) or sm75(T4, 2080Ti) devices, maybe you could try https://github.com/InternLM/lmdeploy