You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While looking through this library I noticed that each time RowScanner.Scan was called a new interface slice was created.
Since the row scanner caches column names and field indexes I wanted to see if there could be a benefit to using a pool of slices rather than allocating a new one each scan.
I created a data struct with 1024 columns and some quick benchmarks to my fork of scany here. The benchmark data struct and the benchmarks are in two new files in my fork, bench_data_test.go and bench_test.go, if anyone wants to run the benchmarks for themselves.
Using a pool of slices reduces the memory usage by over 16000 B/op when scanning into both, a struct or a map. And specifically for a struct, the bytes allocated remain constant even though there are 1024 different columns.
This is a great use of sync.Pool since due to RowScanner's caching the allocated slices are of the same length each time Scan is called. I think it would be useful for RowScanner to provide an option for using a pool instead of allocating a new slice.
The text was updated successfully, but these errors were encountered:
I ran some different benchmarks and noticed caching the column to field index maps would help a lot more when creating a new row scanner each iteration instead of reusing one as what I'm doing here. I created a new issue #25 explaining the cache.
I think creating a new row scanner is a lot more common because of functions like ScanAll/ScanOne.
Thanks for the great library.
While looking through this library I noticed that each time RowScanner.Scan was called a new interface slice was created.
Since the row scanner caches column names and field indexes I wanted to see if there could be a benefit to using a pool of slices rather than allocating a new one each scan.
I created a data struct with 1024 columns and some quick benchmarks to my fork of scany here. The benchmark data struct and the benchmarks are in two new files in my fork,
bench_data_test.go
andbench_test.go
, if anyone wants to run the benchmarks for themselves.Results of benchmarks:
Using a pool of slices reduces the memory usage by over 16000 B/op when scanning into both, a struct or a map. And specifically for a struct, the bytes allocated remain constant even though there are 1024 different columns.
This is a great use of sync.Pool since due to RowScanner's caching the allocated slices are of the same length each time Scan is called. I think it would be useful for RowScanner to provide an option for using a pool instead of allocating a new slice.
The text was updated successfully, but these errors were encountered: