How to get result batch size to match the set batch_size arg in read_arrow_batches

[JChengLinH]

Hello all,
I'm exploring arrow-odbc to try to replace pyodbc that's currently being used due to its speed. I'm trying to get the returned result to match the size I've set for the argument batch_size to read_arrow_batches method but couldn't really get the row numbers returned in a batch to match the set batch_size. I can see from the documentation that the max number of rows upper bound can be dictated by max_bytes_per_batch if it's lower.

However I've set the max_bytes_per_batch to a very high number, eg. 4 times (even tried 100 times just to see if it changed the outcome) the default value and I still cannot get the row numbers returned in a batch to match the set batch_size. For example I've set the batch_size=200000 but only got 176544 rows returned in each batch. Is there a way to get the returned number of rows to exactly match the batch_size?

Thanks a lot in advance, any help is appreciated 😃

[JChengLinH]

Never mind, silly me mixed up max_bytes_per_batch with max_binary_size

[pacman82]

Glad you could figure it out. I hope your experiment will be successful!

[JChengLinH]

Thanks! Really appreciate this wonderful library! It's extremely fast! And you are a very helpful and kind maintainer!