[BUG]: Malloc exception while reading large parquet file

[shamimashik]

Issue Description

Having issues reading large sized checkpoint parquet files.
I'm using code like this -
val results = rowGroupReader.Column(0).LogicalReader<string>().ReadAll(numRows);

Getting the below error -
class parquet::ParquetStatusException (message: 'Out of memory: malloc of size 104478272 failed')

Environment Information

• ParquetSharp Version: [e.g. 1.0.1]
• .NET Framework/SDK Version: [e.g. .NET Framework 4.7.2]
• Operating System: [e.g. Windows 10]

Steps To Reproduce

val results = rowGroupReader.Column(0).LogicalReader<string>().ReadAll(numRows);

Expected Behavior

There should not be any exception

Additional Context (Optional)

No response

[adamreeve]

Hi @shamimashik. Is numRows the total number of rows in the row group? Reading a smaller number of rows at a time might help reduce memory usage. Eg. something like:

const int bufferSize = 1024;
var buffer = new string[bufferSize];
using var columnReader = rowGroupReader.Column(0).LogicalReader<string>();
while (columnReader.HasNext)
{
    var rowsRead = columnReader.ReadBatch(buffer);
    var values = buffer.AsSpan(0, rowsRead);
}

[pathacke]

Hi @adamreeve Thanks for response.

How can we determine an ideal buffer size that works for tables of all sizes? We might have tables with a large number of columns as well as tables with fewer columns. A table with many columns could cause an exception if the buffer size isn't appropriate.

[adamreeve]

The buffer would be used to read one column at a time so the number of columns shouldn't matter. I think you'd need to do testing using your own data to determine a buffer size that works best for you.