hive.max-partitions-per-writers

[Akanksha-kedia]

With default value of hive.max-partitions-per-writers=100 and i m trying to insert into a partitioned table with 2 partition s column i have changed the page chunk with positionCount to be 100(In code 4096) i m getting error"Exceeded limit of 100 open writers for partitions/buckets. I see its using MultiChannelGroupByHash and PagePartitioner but whats the use of this hash table and why differnt writerIndexes is getting created as we shpuld close old ones and start a new writers list accordingly.

@mosabua @wendigo any idea abt this.

Also in case pf huge dataset with large unique partitions, to wat extend we can increase the value of parameter hive.max-partitions-per-writers can we set this value based on unique partitions being set?

Do let me know any more information needed.

[Akanksha-kedia]

Want to get clarity of how each page is getting processed as per code each page of postion 4096 is being processed and if that chunk has unique partitions based on that writeraindexes is returned with partitionsPage() api and indexPage() api which in turn GroupByHashPageIndexer class which is using MultiChannelGroupByHash which is having inner class GetNonDictionaryGroupIdsWork for process and other group add. pagePartitioner.getMaxIndex() can have wat max value ?