Not displaying history at folder layer for some projects

[simranjeetc]

Describe the bug
Clicking on the folder history link throws an error for some repositories

Please refer to the folder structure in the below screenshot

• With the above folder structure, if the history link is clicked on any folder, it gives the above error
• This doesn't happen if the highlighted repository is kept at the parent folder
• This doesn't happen for other repositories i.e. clicking on history folder link for other repositories work just fine. The problem seems to be coming up only with the highlighted repo when it is placed at this nested level.
• The history link works irrespective if it is clicked at the file level

Folder structure

Error

Expected behavior

Clicking on folder history link of the highlighted repo should display the history even in the nested folder

Additional context
Below are the logs

17-Oct-2023 08:57:01.929 FINER [http-nio-8080-exec-8] org.opengrok.indexer.authorization.AuthorizationStack.isAllowed Authorization for "Verity" in "default stack" [required]
17-Oct-2023 08:57:01.929 FINER [http-nio-8080-exec-8] org.opengrok.indexer.authorization.AuthorizationStack.isAllowed Authorization for "Verity" in "default stack" [required] => true
17-Oct-2023 08:57:01.938 FINEST [http-nio-8080-exec-8] org.opengrok.indexer.history.HistoryGuru.getHistoryFromRepository '/opengrok/src/Verity/collection-center' in null is not eligible for history
17-Oct-2023 08:57:01.939 FINE [http-nio-8080-exec-8] org.apache.catalina.core.StandardHostValve.custom Processing ErrorPage[errorCode=404, location=/enoent]
17-Oct-2023 08:57:01.940 FINER [http-nio-8080-exec-8] org.opengrok.indexer.authorization.AuthorizationStack.isAllowed Authorization for "1_verity" in "default stack"

Using the below mentioned docker-compose file to deploy the opengrok image

version: "3"

# More info at https://github.com/oracle/opengrok/docker/
services:
  opengrok-fas:
    container_name: opengrok-fas
    image: opengrok/docker:1.7.11
    restart: always
    volumes:
      - 'opengrok_data_fasa101023:/opengrok/data'
      - '/code/opengrok/src/fas/:/opengrok/src/'  # source code
      - '/code/opengrok/src/ABCCode/:/opengrok/src/ABCCode/' 
      - './etc_fas/:/opengrok/etc/'  # folder contains configuration.xml
      - './conf/:/usr/local/tomcat/conf'
    ports:
      - "9090:8080/tcp"
      - "5001:5000/tcp"
    environment:
      SYNC_PERIOD_MINUTES: '360'
      OPENGROK_LOG_LEVEL: 'INFO'
      NOMIRROR: 'true'
      INDEXER_OPT: '-H -S -R /opengrok/etc/read_only.xml'
    # Volumes store your data between container upgrades
volumes:
  opengrok_data_fasa101023:
networks:
  opengrok-fas:
~                                                                                                                                                                                                           
~                                                                                                                                                                                                           
~                                   

[vladak]

The log message getHistoryFromRepository '/opengrok/src/Verity/collection-center' in null is not eligible for history is the key. The null means that no repository was found for the path. This means when the project was added, the repository scan did not discover the repository for some reason.

[vladak]

The scanningDepth indexer tunable affects how deep into the directory structure under source root repositories will be searched for. That said, /opengrok/src/Verity/collection-center is 2 levels below source root so the default (2) should work.

[simranjeetc]

Right. There are approx 100 other repos in that folder. None of them show that problem. The problem comes only when this repo is placed here. Even this repo works fine when placed in the parent folder

[vladak]

I just tried repository scan of a structure resembling your use case (/var/opengrok/src.scantest/foo/bar where /var/opengrok/src.scantest is the source root, foo is directory with the bar being its sole subdirectory. bar is a Git repo.) with the latest OpenGrok and it works in the default configuration - both when running the indexer by hand (with -S -P -H etc.) and when using the API to add a project with its repositories.

According to the docker-compose configuration you are running 1.7.11 which is before the scanning depth normalization change was done in PR #4148, I wonder if this could be related, however at this point this is just a blind shot.

[vladak]

Also, adding -R to the indexer options will likely cause some trouble. The READONLY_CONFIG_FILE environment variable should be used instead. Similarly, adding -H and -S will probably not have the desired effect.

[simranjeetc]

Ok. Will make those changes. Can you tell how I can include an indexer configuration like scanningDepth in the docker compose file shared above?

[vladak]

I am not sure it will help but maybe worth trying.

Create read-only configuration file and let the container now about it using the READONLY_CONFIG_FILE environment variable.

The contents of the file will look like this (I haven't tried):

<?xml version="1.0" encoding="UTF-8"?>
<java version="1.8.0_121" class="java.beans.XMLDecoder">
 <object class="org.opengrok.indexer.configuration.Configuration">

  <void property="scanningDepth"> 
   <int>5</int> 
  </void>

 </object>
</java>

[simranjeetc]

And is there a way I can knock off the index and history data entry only for the repository in question and force rebuilt?

[vladak]

In the official Docker image it's all or nothing. In OpenGrok itself, one can reindex one project, however not a repository for a project but again, not in the Docker image. That said, if the mirroring is on, the per-project reindex in Docker will check whether there are any incoming changes and will not start reindex if not. Also, the incremental nature of the reindex for some repository types (Git, Mercurial) make the reindex very lightweight, depending on the amount of newly added changes of course.

[simranjeetc]

logs.txt
Updated to latest docker image
Attaching the logs. It shows some exceptions. May be it is helpful for you to figure out whats happening

[vladak]

The ls command only matches subdirectories. For some basic idea how many Git repositories are contained within the project that could be reached by OpenGrok you'd have to run something like find /opengrok/src/Verify/ -maxdepth 3 -type d -name '.git' or such.

[simranjeetc]

logs-2010.txt

Please find the logs with DEBUG flag turned on. This opengrok image has addressed the underlying issue. I see that I am able to see the history of the repository at directory level

[simranjeetc]

Is there a setting to render README.md files? They used to be rendered by default till a few versions back. Now they are being shown just as code.

[vladak]

Is there a setting to render README.md files? They used to be rendered by default till a few versions back. Now they are being shown just as code.

That's probably a regression. Please create a new issue for that.

[simranjeetc]

Sure. Will do that