Best approach to create backup DR instance of opengrok

[jkjha]

We have an operational OpenGrok production instance (Docker version 1.12.12+) successfully running in one data center. Currently, we're working on creating a disaster recovery (DR) instance in a different data center (DC). The total index size is approximately 5+ terabytes, including all data.

Initially, our plan was to use the project list from the production system to clone the projects and reindex everything from scratch in the new DC. However, due to network latency issues between the new DC and the source code repository, we've encountered the following problem: #747

The issue arises when the indexer attempts to run /usr/bin/p4 filelog -sti ..., but it times out, and it begins collecting historical information for each individual file.

Here's a timestamp for reference:

2023-10-10 06:47:17.074+0000. A warning message indicates that the process execution of the command [/usr/bin/p4 filelog -sti ...] in the directory '/opengrok/src/<perforce_project_name>' was terminated due to a timeout after 600 seconds.

The Perforce repository in question is around 6 gigabytes, and this results in a large number of Perforce requests, causing a heavy load on the Perforce server and complaints from the server administrator.

To address this issue, we attempted to add the following configurations one by one to the read-only.xml file, but unfortunately, none of these solutions worked:

  <void property="fetchHistoryWhenNotInCache">
   <int>false</int>
  </void>
  <void property="indexerCommandTimeout">
   <int>10800</int>
  </void>

When comparing the number of Perforce requests between our production instance (less than 100) and the new data center instance with known network latency issues (almost half a million), there's a significant difference.

So, we have two questions:

Do you have any suggestions to overcome this issue?
Is it possible to copy the source code data from our production system to the new virtual machine in the different data center and run OpenGrok from there? We anticipate challenges related to conflicting source code and index synchronization. However, given our requirements, can you please advise if this approach could work?

[vladak]

I'd focus on small things first, like: why is the indexerCommandTimeout set in read-only configuration not honored ? Looking into the PerforceHistoryParser class source, all the commands are invoked via Executor(cmd, file) which means that the indexerCommandTimeout should be honored. This is not as clear in the PerforceRepository class which is using Executor as well. That said, the filelog p4 command is executed via PerforceHistoryParser#parseDirectory() which should use the indexerCommandTimeout so setting the timeout to non-default value should just work. How do you run the indexer ?

[jkjha]

Since its docker instance, it runs from sync.yml : https://github.com/oracle/opengrok/blob/master/docker/sync.yml
with slight modification in command field, here is the modified sync.yml:

- command:
    args: [opengrok-reindex-project,
      --api_timeout, '%API_TIMEOUT%',
       -J=-XX:-UseGCOverheadLimit, -J=-Xmx32g, -J=-server,
       --jar, /opengrok/lib/opengrok.jar,
       -t, /opengrok/etc/logging.properties.template, -p, '%PROJ%', -d, /opengrok/log/%PROJECT%,
       -U, '%URL%', -P, '%PROJECT%', --,
      --connectTimeout, '%API_TIMEOUT%',
      -r, dirbased, -G, -m, '32768', --leadingWildCards, 'on',
      -c, /usr/local/bin/ctags, -U, '%URL%', -o, /opengrok/etc/ctags.config, -H, '%PROJECT%']
    limits: {RLIMIT_NOFILE: 8192}
    args_subst: {"%API_TIMEOUT%": "$API_TIMEOUT"}

Here is the first few lines of indexer log:

2023-10-10 06:36:41.469+0000 FINE t1 Executor.registerErrorHandler: Installing default uncaught exception handler
2023-10-10 06:36:41.767+0000 INFO t1 Configuration.read: Reading configuration from /tmp/tmpr1xjtym3
2023-10-10 06:36:41.891+0000 INFO t1 Indexer.parseOptions: Indexer options: [-A, .vmodl:JavaAnalyzer, -R, /tmp/tmpr1xjtym3, --connectTimeout, 300, -r, dirbased, -G, -m, 32768, --leadingWildCards, on, -c, /usr/local/bin/ctags, -U, http://localhost:8080/, -o, /opengrok/etc/ctags.config, -H, <perforce_project_name>]

I can see indexerCommandTimeout is not passed from above log... however it exists on read-only.xml and I restarted container after adding this field.

 <void property="sourceRoot">
   <string>/opengrok/src</string>
  </void>
  <void property="interactiveCommandTimeout">
   <int>300</int>
  </void>
  <void property="fetchHistoryWhenNotInCache">
   <int>false</int>
  </void>
  <void property="indexerCommandTimeout">
   <int>10800</int>
  </void>

[vladak]

The path to the read-only configuration file has to be set via the READONLY_CONFIG_FILE environment variable in the container. When the container starts and recognizes this variable to be set, it will merge it with the current configuration in configuration.xml. The container will emit the Merging read-only configuration ... log message. It will move the resulting file to configuration.xml. Does your configuration.xml contain the above tunables ?

opengrok-reindex-project retrieves the configuration from the running web app and uses it as read-only configuration. That's why you can see a temporary file name with the -R option. You can check the value of the tunables in the web app like so:

curl http://localhost:8080/source/api/v1/configuration/indexerCommandTimeout

The same API endpoint can be used for setting tunable values: https://opengrok.docs.apiary.io/#reference/0/configuration-field