Multi-DC (V3) replication from older versions

[jches-sg]

Hi everyone! My team is in the planning stages of migrating some riak clusters that run on an old basho riak-ee version 2.0.9, and we're exploring options for adopting the newer open source releases. After experimenting with riak-repl, we've found that a cluster on 2.0.9 can talk to some newer versions, but even for example a cluster running 2.2.6 does not persist any of the objects it receives during replication, and we get a lot of warning messages in the sink cluster's logs saying:

Remote and local bucket properties differ for type <<"some_bucket_type">>

The only bucket property I'm seeing that is different between the two clusters is the node_confirms field, which is not present on the old version (that feature was added with the 2.2.5 release as I understand it). Could that field be the underlying issue here? I only have rudimentary knowledge of erlang so it's not clear to me whether this bucket_props_match function is comparing all bucket props or just the bucket "datatype" field https://github.com/OpenRiak/riak_repl/blob/openriak-3.4/src/riak_repl_bucket_type_util.erl#L21

Assuming that it is comparing all props, this essentially means multi-DC replication can't work between older versions any any of the open source releases. Since it doesn't look like the node_confirms feature can be turned off (short of building a patched version from source), would the best approach for migrating from an older version be to do an in-place upgrade?

Thanks in advance for any insight, ideas, or feedback on this.

[martinsumner]

Have you followed the instructions here?

https://docs.riak.com/riak/kv/latest/configuring/v3-multi-datacenter/index.html#default-bucket-properties

I don't think node_confirms matters as it is not a property checked in the hash - it only checks the properties in the list. However, the list changed in 2.2.0, so the instructions above need to be followed to make the list consistent between the clusters.

[jches-sg]

Aha, thank you! This is almost certainly it; I hadn't seen that note (and now that you pointed it out I can't believe I hadn't noticed it).

[jches-sg]

Yep I confirmed this works with that config change. Thanks again for taking time to answer!