Can one SCM support multiple OM HAs?

[ivandika3]

Hi, I'm currently exploring the possibilities of allowing multiple OM HAs to share one SCM (and the Datanodes it manages). From what I currently understand, SCM should technically be able to support multiple OM HAs since its responsibility is as a block space manager. Hence it should not matter whether there are multiple OM HAs (i.e. SCM clients). Are there any possibilities of conflicts between the OM HAs or things I should look out for? Thanks in advance.

[devmadhuu]

Do you mean OM HA ? In a cluster, I have seen multiple OMs in which one is leader and other followers, not sure what you meant by "SCM support multiple OMs"

[ivandika3]

I updated the question and description to clarify this.

[devmadhuu]

Ok, what is the business use case for multiple OM HAs ? why OM HAs where 1 is leader and other followers not sufficient ?

[ivandika3]

In our cluster, we seem to have some bottlenecks on the OM side, whereas the SCM side is able to handle the load. Hence, the next time we spin up a new Ozone cluster we can instead just spin up the new OM HA to point to the existing SCM.

[devmadhuu]

What kind of bottlenecks you are pointng here for OM, IMO those were mostly in HDFS NN due to lot of blocks processing. however in Ozone, we have not seen such bottlenecks.

[ivandika3]

Under high load, we see that sometimes OM Hadoop RPC queue length and processing time spiked which increased the latency of some user operations.

[kerneltime]

@ivandika3 Architecturally, the separation of front-end load and back-end load does allow for what you ask, but it would need significant code changes in the client to have a unified view across OMs.
Before as a community, we start a project to shard OM, I would like to know how what your current config for OM is and how we can help push that performance further.

[ivandika3]

@kerneltime Thank you for addressing this.

Architecturally, the separation of front-end load and back-end load does allow for what you ask, but it would need significant code changes in the client to have a unified view across OMs.

Currently, the unified view across OMs is not required, we can configure a different client for each OM HA service. The main worry is whether it is safe (durability-wise and consistency-wise) to share an SCM (e.g. can the same container shared between different OMs , will the container cache be affected, etc). I have written some integration testing for multiple OM HA setup, and currently no issue is detected.

I would like to know how what your current config for OM is and how we can help push that performance further.

Thank you for the help. We can discuss this in the future.

[swagle]

I have written some integration testing for multiple OM HA setup, and currently, no issue is detected.

That is great to hear @ivandika3. Do you have a pull-request for this test?

Ozone was architected to support this at the protocol level. However, making it a unified namespace solution requires solving non-unique challenges including but not limited to sharding, hot-spotting, cross-shard consistency based on sharding strategy (desirable for a GA solution), and no data movement technically needed (all metadata level). We do have tables like user, S3 secrets, and ACLs which do not cleanly separate at volume and bucket levels (possible shard candidates), Also, bucket linking across volumes.

[kerneltime]

@ivandika3 that is super interesting. The HDDS layer in the architecture (SCM + Datanode) should not depend on OM's decisions. This is a worthy experiment to run and test. If a single namespace is unnecessary and OM's can operate as independent entities, you gain by pooling the capacity across multiple namespaces, which has value. Feel free to let us know how we can help.

[ivandika3]

Thank you @kerneltime and @swagle for further clarifications. I will run further experiments on this.