[BUG] Upgraded indices aren't considered compatible for upgrade to 3.0.0

[ShowMeTheGita]

Describe the bug

Also available here

---

Versions):
OpenSearch 2.19.2
OpenSearch 3.0.0

Describe the issue:
During an upgrade from OpenSearch 2.19.2 to 3.0.0, we're encountering compatibility errors with system indices (primarily .kibana_* indices) that were originally created in Elasticsearch 7.10.2 but have been upgraded to OpenSearch 2.19.0 at some point in the past:

The index [[.kibana_xxx/uuid]] was created with version [7.10.2] but the minimum compatible version is [2.0.0]. It should be re-indexed in OpenSearch 2.x before upgrading to 3.0.0.

Checking the _settings?human for that index returns the following:

        "version": {
          "upgraded": "136407827",
          "created_string": "7.10.2",
          "upgraded_string": "2.19.0",
          "created": "7100299"
        }

Is it expected behavior that system indices (.kibana_*, etc.) that were originally created in Elasticsearch 7.x (but upgraded since) would cause compatibility issues during an upgrade? These indices contain configuration data and are usually managed by OpenSearch.

The upgrade documentation index compatibility reference ends in OpenSearch versions 2.13.0/2.14.0 which suggests that anything above that would likely be compatible with 3.0.0, or is this a misinterpretation?

The indices function correctly in OpenSearch 2.19.2 and contain the upgraded_string: "2.19.0" field.

Could it be that this validation is only looking at created_string and not upgraded_string? Is this expected behavior?
I'm afraid I couldn't find much documentation specifying the impact of these attributes inside version related to the upgrade process.

I'm still in the process of developing an automated workaround for this, but so far it has involved doing the following steps:

• Create a temporary index with the same template as the original
• Reindex the original index to the temporary index
• Delete the original index
• Create a new index with the same original index name and template
• Reindex from the temporary index back to the new one

EDIT: From the reindexing process mentioned above we came to understand a couple of things:

• The old indexes that existed with incremental suffixes such as _1 and _2 seem to have been created during other upgrade activities. We confirmed that they were simply old aliases that were no longer being used. After this finding, we were able to simply delete them and only keep the last _* index which indeed was aliased to the same index name, but without the suffix
• Reindexing by itself was not enough because after it is complete, the new index does not contain the alias as the original one. To fix this, we simply added the alias back after the reindexing was completed
• All of the above being said, we still had internal indexes from plugins such as .opendistro_alerting that did require a reindexing even though they did not have an alias. Since we do not use most of those plugins, we went ahead with the rendexing and it didn't seem to break any functionality. Still, our showstopper as of right now is the .opendistro_security as we do not know what the outcome will be of reindexing this sensitive index. We're consdering reindexing it after having a security backup using the securityadmin.sh script. Our main concern is the reindexing process breaking something and not being able to even login as admin user

I've added the link to this other forum post below, but want to underline this since it does seem to be the same problem

Do be aware that this is just a proof of concept, but doing so seems to not have broken any functionality, and the new index settings now shows the following:

        "version": {
          "created_string": "2.19.2",
          "created": "136408027"
        }

... which I'm hopeful will fix the issue.

However, two questions remain:

• Is this a bug or expected behavior? Should upgraded indices (system or otherwise) not be considered compatible?
• Are system indices also supposed to be reindexed to match the compatibility reference table constraints? I ask because it always raises some suspicions when these indices need to be tampered with at all.

Thanks.

---

Relevant links
Impossible upgrade Opensearch 1.3.2 to 2.0.0 - index version incompatible

Related component

Other

To Reproduce

1. Upgrade an incompatible index to a newer version
2. Attempt to upgrade from 2.X,X to major 3.X.X

Expected behavior

Upgraded indices are considered compatible according to index compatibility reference

Additional Details

Plugins
n/a

Screenshots
n/a

Host/Environment (please complete the following information):

• OS: RHEL
• Version 7.9

Additional context
n/a

[faabsen]

Got the same problem for the .opendistro_security index. Interestingly, it seems like the index did not got upgraded correctly to 2.19.2

        "version": {
          "upgraded": "136367827",
          "created_string": "1.2.4",
          "upgraded_string": "2.15.0",
          "created": "135238227"
        }

[ShowMeTheGita]

@faabsen same here - currently the .opendistro_security is the only index left to upgrade.
We have managed to figure out what the issue was related to the other ones, I'll update the original description with additional info

[arthar360]

Same problem with ".opendistro-anomaly-detection-state"

{
  ".opendistro-anomaly-detection-state": {
    "settings": {
      "index": {
        "creation_date_string": "2021-11-30T09:14:58.621Z",
        "hidden": "true",
        "number_of_shards": "1",
        "provided_name": ".opendistro-anomaly-detection-state",
        "creation_date": "1638263698621",
        "number_of_replicas": "1",
        "uuid": "VB38MmyG-HSKDug",
        "version": {
          "upgraded": "136327927",
          "created_string": "1.1.0",
          "upgraded_string": "2.11.1",
          "created": "135227827"
        }
      }
    }
  }
}

I tried to reindex as admin user. All I get is permission denied. Tried applying all_access role etc. to the user but no luck.

[pawelw1]

@arthar360 Try with superadmin certificate defined in admin_dn. Most of the hidden indices requires that certificate.

[ShowMeTheGita]

We've successfully upgraded to 3.0.0 so I'll leave our findings here:

• When it comes to system indexes (e.g. .opendistro*, .opensearch*) causing issues, reindexing them to a temporary index and back to the index with the original name did work - with the minor consideration that it's important to take note if any alias exists before the reindexing process ends so that it can be re-aliased afterwards
• As already mentioned on the description edit, system indexes with incremental suffixes in their name (e.g. _1) were remnants from previous opensearch upgrades. The key to pushing past these was to query GET .index-name*/_alias and correctly identifying which index was the one with an alias pointing to the index with the same name, without the suffix - this was usually the one with the highest number in the suffix, and that would be the one to reindex and not delete. If there's an index with incremental suffixes up until _6, for example, querying the _alias endpoint with a wildcard will likely confirm that indexes _1 to _5 do not point to any index anymore and are safe to be tagged for deletion - only _6 would need to be reindexed and re-aliased after
• .opendistro_security was trickier because it requires super admin privileges to run anything against, so what we did was run the securityadmin.sh script to first backup all the info into a yaml file, then use the same tool to delete the index, and again a third time to recreate it. Detailed steps below based on the official documentation page containing usage examples

--> Step 1 - Backup all security configurations 
bash /opt/opensearch-2.19.2/plugins/opensearch-security/tools/securityadmin.sh \
-backup /opt/opensearch-2.19.2/plugins/opensearch-security/tools/bkp \
-icl \
-nhnv \
-ts /opt/opensearch-2.19.2/config/SSL/truststore.jks \ 
-tspass <password> \
-ks /opt/opensearch-2.19.2/config/SSL/keystore.jks \ 
-kspass <password> \
-h <hostname>

--> Step 2 - Delete security configurations
bash /opt/opensearch-2.19.2/plugins/opensearch-security/tools/securityadmin.sh \
-dci \
-icl \
-nhnv \
-ts /opt/opensearch-2.19.2/config/SSL/truststore.jks \ 
-tspass <password> \
-ks /opt/opensearch-2.19.2/config/SSL/keystore.jks \ 
-kspass <password> \
-h <hostname>

--> Step 3 - Restore/recreate all security configurations
bash /opt/opensearch-2.19.2/plugins/opensearch-security/tools/securityadmin.sh \
-cd /opt/opensearch-2.19.2/plugins/opensearch-security/tools/bkp \
-icl \
-nhnv \
-ts /opt/opensearch-2.19.2/config/SSL/truststore.jks \ 
-tspass <password> \
-ks /opt/opensearch-2.19.2/config/SSL/keystore.jks \ 
-kspass <password> \
-h <hostname>

Note: We manually edited the securityadmin.sh script to remove piping of output to /dev/null in order to better troubleshoot connectivity/certificate/keystore issues

If for any reason you do not have a certificate which is accepted by the dn on plugins.security.authcz.admin_dn let it be known that a temporary workaround for this can be to add a new entry to the above configuration on the opensearch,yml of all nodes and then perform a rolling restart on the cluster. There was a piece of documentation somewhere outlining the need for this configuration to be the same on all nodes in order to work.

My two cents is that this may have likely been caused by the usage of opensearch.http.override_main_response_version=true on previous upgrades. Seems like this configuration just pushed this inevitable problem up until an upgrade to a new major was required.

This still doesn't explain the main reason for raising this issue - OpenSearch apparently validating the index version against created_string and not upgraded_string on startup, causing the upgrade to fail.

[AndreKurait]

If you are able to tolerate moving to a new cluster, you can use the Migration Assistant to move and upgrade clusters to OS 3 https://docs.opensearch.org/docs/latest/migration-assistant/

[krisfreedain]

Catch All Triage - 1 2

@andrross - can you have someone take a look at this?

[arthar360]

Update:

Thanks for the steps @ShowMeTheGita.

1. I manually reindexed all indices from 1.X to 2. Except one which I forgot.
2. During upgrade opensearch complained it cannot proceed until I upgrade this index.
3. I then started the cluster in opensearch 2 and reindexed.
4. No indices were listed in GET /*/_settings?human which showed version 1.X.
5. Started the upgrade again. It went fine for sometime
   Then it ends up with

.kibana_92668751_admin_1/5b5LfzhsTxxYkyJEYnsmUg] version not supported: 1.1.0 minimum compatible index version is: 2.0.0   

6. So now I switched back to opensearch 2 but this time it complained about lunace version mismatch and did not start the cluster.

7. So neither I could upgrade to opensearch 3, nor downgrade to 2.

8. I tried deleting kibana_92668751_admin_1 index on nodes manually, restart nodes etc. but it did not help.

Since it was out production cluster, the fastest way to get it up and running was to recreate a new cluster and restore from snapshot.

I don't understand if opensearch stop upgrade like it did in step 2 then why did it upgrade lunace version of other indices and then failed on step 5.

[jkrishnarao2003]

I am also facing the same issue when I am upgrading from 2.18.0 to 3.1.0. What I don't understand is when all the indexes are with upgraded version 2.18.0 even though they are created with version 1.3.0 why OpenSearch upgrade process still checking for created version instead of upgraded version? Is this really a bug?

[ShowMeTheGita]

@arthar360 unfortunately we initially faced the same issue.
It does state in the documentation that once you perform the upgrade you cannot rollback as any indexes created with 3.* will be incompatible with older versions.

Our solution was to simply delete 1-2 indexes that were created within the 24h (when they rolled over after upgrading one node) after we upgraded to 3 and then we were able to successfully rollback the cluster with minimal data loss. So you do need to keep in mind if any indexes will rollover or are created during the upgrade process, because if you upgrade one node do version 3 and an index is created in version 3, you will no longer be able to rollback until it is deleted.

[ShowMeTheGita]

What I don't understand is when all the indexes are with upgraded version 2.18.0 even though they are created with version 1.3.0 why OpenSearch upgrade process still checking for created version instead of upgraded version? Is this really a bug?

@jkrishnarao2003

That's the entire point of this issue. I have the same understanding that they should be compatible from reading the documentation, but it seems that's not the case. An upgraded index does not seem to meet the criteria for compatibility even if it is on a compatible version if it was initially created with an incompatible version - this is what trial and error has shown