Spark + Ozone 1.4.0 Factor replication

[yazynin]

I set the replication factor to 1

ozone sh bucket info applications/jupyter { "metadata" : { }, "volumeName" : "applications", "name" : "jupyter", "storageType" : "DISK", "versioning" : false, "listCacheSize" : 1000, "usedBytes" : 374859618074, "usedNamespace" : 675880, "creationTime" : "2024-01-19T08:16:24.020Z", "modificationTime" : "2024-08-01T09:55:34.670Z", "sourcePathExist" : true, "quotaInBytes" : -1, "quotaInNamespace" : -1, "bucketLayout" : "FILE_SYSTEM_OPTIMIZED", "owner" : "xxxx", "replicationConfig" : { "replicationFactor" : "ONE", "requiredNodes" : 1, "replicationType" : "RATIS" }, "link" : false }

But spark create files with factor replication = 3

2024-08-06 10:38:30,980 | INFO | OMAudit | user=xxxx | ip=x.x.x.x | op=CREATE_FILE {volume=applications, bucket=jupyter, key=staging-dir/xxxx/.sparkStaging/application_1721203053959_3728/com.microsoft.cntk_cntk-2.4.jar, dataSize=268435456, replicationType=RATIS, **replicationFactor=THREE**} | ret=SUCCESS |
How fix it?

[errose28]

Hi @yazynin, the replication config for the bucket is a default if the client does not provide one. According to the audit log, it looks like the spark client explicitly requested to create a key of replication factor 3, which overrides the bucket default. Check if your spark client has the configuration spark.hadoop.dfs.replication set to 3, and change it to 1 if that is the desired behavior. Ozone has client side replication configs as well: ozone.replication (you can set this to 1) and ozone.replication.type (you can set this to RATIS). I believe the Ozone client configs will not be used if the layer above, like spark invoking Ozone through ofs, has already specified a replication config.

[yazynin]

Hi @errose28 yes I already set spark.hadoop.dfs.replication = 1 in spark init, but .sparkStaging files created with replicationFactor=THREE

[yazynin]

Ozone connected as ofs

[yazynin]

looks like only this what matter:

<property><name>ozone.replication</name><value>1</value></property>
<property><name>ozone.replication.type</name><value>RATIS</value></property>

spark.yarn.submit.file.replication and spark.hadoop.dfs.replication is ignored