Support for Fcrepo 6 in Valkyrie (#6976)

* Convert Ldp::Gone to ObjectNotFoundError

Ldp::Gone is the error raised by the fedora adapter when the resource is a tombstone after being deleted.

* Use the real metadata adapter

* Ensure embargo/lease factory is using DateTime

The valkyrie fedora adapter does not convert RDF Date causing comparison failures.
In real use the embargo transaction converts to a datetime, but this factory sets the value directly, so let's make it match real use.

* Avoid using << on frozen default array in specs

* Sometimes the original file is not first in file_ids

* Convert Ldp::NotFound to Valkyrie::Persistence::ObjectNotFoundError

This fixes file_set_indexer_spec which has a bunch of odd setup and mocking, likely from before our factories were equipped to handle building work/file_set/file_metadata stacks. However, it feels right to include here and be treated as an ObjectNotFoundError.

* Fix specs stubbing Hyrax::TimeService with a string

* More date format spec fixes

* Limit rails versions to speed dependency resolution

* Bump valkyrie, use main until new release

* Unpin faraday, rdf 3.3.2 fixed the bug

* Add sirenia to Github Actions CI; Significant rework of docker compose

- All 3 test apps use the same container image.
- dassie stays in hyrax-webapp, koppie/sirenia live in hyrax-koppie.
- db_migrate service moved to dev-entrypoint in web service.
- Worker waits to start until rails app is up, should reduce bundle install racing.
- hyrax-engine-dev target now based on hyrax-worker-base.
- Chrome is still a pain and randomly stops responding.

* rubocop

* Docker and GHA fixes

Add worker to depends_on to avoid volume copy race.
Fix Dockerfile style issues
Align chrome service config
Corrected container names for log capture

* Relax class matcher strictness

Now that this spec uses the real valkyrie adapter,
the object coming out of wings is really an anonymous
descendant of the valkyrie model class.

* Restore `rake assets:precompile` for javascript tests

* Publish deletion first to avoid encountering tombstones

* Use correct rails root

* rubocop

* Convert custom queries added for statistics to solr

* Optimize admin set list (also fixes sorting issues)

* Sort the admin sets for real life

* :clean_repo for flaky specs

* Bump valkyrie version

* Rename db-wait.sh script to service-wait.sh

* Return Resources instead of SolrHits

* Undo running koppie/sirenia from hyrax-koppie dir

* We are not yet compatible with v1 of openseadragon gem

Author: dlpierce

.dassie/.env

@@ -2,7 +2,7 @@ ANALYTICS_START_DATE=2021-08-21
 BUNDLE_GEMFILE=Gemfile.dassie
 BUNDLE_PATH=/app/bundle
 CH12N_TOOL=fits_servlet
-CHROME_HEADLESS_MODE=false
+CHROME_HEADLESS_MODE=true
 DATABASE_CLEANER_ALLOW_REMOTE_DATABASE_URL=true
 DATABASE_TEST_URL=postgresql://hyrax_user:hyrax_password@postgres/hyrax_test?pool=5
 DATABASE_URL=postgresql://hyrax_user:hyrax_password@postgres/hyrax?pool=5

.dassie/Gemfile

@@ -30,7 +30,7 @@ gem 'jbuilder', '~> 2.5'
 gem 'jquery-rails'
 gem 'pg', '~> 1.3'
 gem 'puma'
-gem 'rails', '~> 6.1'
+gem 'rails', '~> 6.1', '>= 6.1.7.10'
 gem 'riiif', '~> 2.1'
 gem 'rsolr', '>= 1.0', '< 3'
 gem 'sass-rails', '~> 6.0'

.dockerignore

@@ -14,9 +14,11 @@ artifacts/*
 coverage/*
 chart/*
 
-Gemfile.lock
-.dassie/Gemfile.lock
-.koppie/Gemfile.lock
+Gemfile*lock
+*/Gemfile*lock
 
-node_modules/*
+
+node_modules
+*/node_modules
 yarn.lock
+*/yarn.lock

.github/workflows/lint-build-test.yml

@@ -1,5 +1,5 @@
 name: 'Lint Build Test'
-run-name: Link Build Test of ${{ github.ref_name }} by @${{ github.actor }}
+run-name: Lint Build Test of ${{ github.ref_name }} by @${{ github.actor }}
 on:
   push:
     branches:
@@ -44,39 +44,35 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
-      matrix:
-        ci_test_app: [dassie,koppie]
     steps:
       - name: Checkout
         uses: actions/checkout@v4
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
       - name: Build and export
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
-          build-args: |
-            APP_PATH=.${{ matrix.ci_test_app }}
           cache-from: type=gha
           cache-to: type=gha, mode=max
           context: .
           target: hyrax-engine-dev
-          tags: samvera/${{ matrix.ci_test_app }}:latest,samvera/${{ matrix.ci_test_app }}:${{ github.sha }}
-          outputs: type=docker,dest=/tmp/${{ matrix.ci_test_app }}-${{ github.sha }}.tar
+          tags: samvera/hyrax-dev:latest,samvera/hyrax-dev:${{ github.sha }}
+          outputs: type=docker,dest=/tmp/hyrax-dev-${{ github.sha }}.tar
       - name: Upload built image artifact
         uses: actions/upload-artifact@v4
         with:
-          name: ${{ matrix.ci_test_app }}-image
-          path: /tmp/${{ matrix.ci_test_app }}-${{ github.sha }}.tar
+          name: hyrax-dev
+          path: /tmp/hyrax-dev-${{ github.sha }}.tar
 
   test:
     needs: build
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
-        ci_test_app: [dassie,koppie]
-        ci_node_total: [8]
-        ci_node_index: [0,1,2,3,4,5,6,7]
+        ci_test_app: [dassie,koppie,sirenia]
+        ci_node_total: [4]
+        ci_node_index: [0,1,2,3]
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -89,11 +85,11 @@ jobs:
       - name: Download built image artifact
         uses: actions/download-artifact@v4
         with:
-          name: ${{ matrix.ci_test_app }}-image
+          name: hyrax-dev
           path: /tmp
       - name: Start containers
         run: |
-          docker load --input /tmp/${{ matrix.ci_test_app }}-${{ github.sha }}.tar
+          docker load --input /tmp/hyrax-dev-${{ github.sha }}.tar
           docker image ls -a
           docker compose -f docker-compose-${{ matrix.ci_test_app }}.yml up -d --quiet-pull --pull missing --no-build
       - name: RSpec
@@ -107,7 +103,7 @@ jobs:
         if: always()
         uses: jwalton/gh-docker-logs@v2
         with:
-          images: 'seleniarm/standalone-chromium'
+          images: 'selenium/standalone-chromium,postgres,fcrepo/fcrepo,solr,bitnami/redis,ghcr.io/samvera/fitsservlet,ghcr.io/samvera/fcrepo4'
       - name: Move Test Files
         if: always()
         env:

.github/workflows/test-results.yml

@@ -1,5 +1,5 @@
 name: Test Results
-
+run-name: Test Results of ${{ github.ref_name }} by @${{ github.actor }}
 on:
   workflow_run:
     workflows: ["Lint Build Test"]

.koppie/.env

@@ -2,7 +2,7 @@ APP_NAME=koppie
 BUNDLE_GEMFILE=Gemfile.koppie
 BUNDLE_PATH=/app/bundle
 CH12N_TOOL=fits_servlet
-CHROME_HEADLESS_MODE=false
+CHROME_HEADLESS_MODE=true
 DATABASE_CLEANER_ALLOW_REMOTE_DATABASE_URL=true
 DATABASE_TEST_URL=postgresql://hyrax_user:hyrax_password@postgres/hyrax_test?pool=5
 DATABASE_URL=postgresql://hyrax_user:hyrax_password@postgres/hyrax?pool=5
@@ -29,7 +29,7 @@ POSTGRES_PASSWORD=hyrax_password
 POSTGRES_USER=hyrax_user
 RACK_ENV=development
 RAILS_ENV=development
-RAILS_ROOT=.koppie
+RAILS_ROOT=/app/samvera/hyrax-webapp
 RAILS_LOG_TO_STDOUT=false
 RAILS_QUEUE=sidekiq
 REDIS_HOST=redis

.koppie/Gemfile

@@ -30,7 +30,7 @@ gem 'jbuilder', '~> 2.5'
 gem 'jquery-rails'
 gem 'pg', '~> 1.3'
 gem 'puma'
-gem 'rails', '~> 6.1'
+gem 'rails', '~> 6.1', '>= 6.1.7.10'
 gem 'riiif', '~> 2.1'
 gem 'rsolr', '>= 1.0', '< 3'
 gem 'sass-rails', '~> 6.0'

.koppie/scripts/db-migrate-seed.sh

@@ -1,7 +1,7 @@
 ARG ALPINE_VERSION=3.19
-ARG RUBY_VERSION=3.2.4
+ARG RUBY_VERSION=3.2.6
 
-FROM ruby:$RUBY_VERSION-alpine$ALPINE_VERSION as hyrax-base
+FROM ruby:$RUBY_VERSION-alpine$ALPINE_VERSION AS hyrax-base
 
 ARG DATABASE_APK_PACKAGE="postgresql-dev"
 ARG EXTRA_APK_PACKAGES="git"
@@ -49,7 +49,7 @@ ENTRYPOINT ["hyrax-entrypoint.sh"]
 CMD ["bundle", "exec", "puma", "-v", "-b", "tcp://0.0.0.0:3000"]
 
 
-FROM hyrax-base as hyrax
+FROM hyrax-base AS hyrax
 
 ARG APP_PATH=.
 ARG BUNDLE_WITHOUT="development test"
@@ -59,7 +59,7 @@ ONBUILD RUN bundle install --jobs "$(nproc)"
 ONBUILD RUN RAILS_ENV=production SECRET_KEY_BASE=`bin/rake secret` DB_ADAPTER=nulldb DATABASE_URL='postgresql://fake' bundle exec rake assets:precompile
 
 
-FROM hyrax-base as hyrax-worker-base
+FROM hyrax-base AS hyrax-worker-base
 
 USER root
 RUN apk --no-cache add bash \
@@ -78,10 +78,10 @@ RUN mkdir -p /app/fits && \
     sed -i 's/\(<tool.*TikaTool.*>\)/<!--\1-->/' /app/fits/xml/fits.xml
 ENV PATH="${PATH}:/app/fits"
 
-CMD bundle exec sidekiq
+CMD ["bundle", "exec", "sidekiq"]
 
 
-FROM hyrax-worker-base as hyrax-worker
+FROM hyrax-worker-base AS hyrax-worker
 
 ARG APP_PATH=.
 ARG BUNDLE_WITHOUT="development test"
@@ -91,40 +91,19 @@ ONBUILD RUN bundle install --jobs "$(nproc)"
 ONBUILD RUN RAILS_ENV=production SECRET_KEY_BASE=`bin/rake secret` DB_ADAPTER=nulldb DATABASE_URL='postgresql://fake' bundle exec rake assets:precompile
 
 
-FROM hyrax-base as hyrax-engine-dev
+FROM hyrax-worker-base AS hyrax-engine-dev
 
-USER root
-RUN apk --no-cache add bash \
-  ffmpeg \
-  mediainfo \
-  perl
 USER app
-
-ARG APP_PATH=.dassie
 ARG BUNDLE_WITHOUT=
+ENV HYRAX_ENGINE_PATH=/app/samvera/hyrax-engine
 
-ENV HYRAX_ENGINE_PATH /app/samvera/hyrax-engine
-
-COPY --chown=1001:101 $APP_PATH /app/samvera/hyrax-webapp
+COPY --chown=1001:101 .dassie /app/samvera/hyrax-webapp
 COPY --chown=1001:101 . /app/samvera/hyrax-engine
 
 RUN bundle -v && \
-  bundle install --jobs "$(nproc)" && \
-  cd $HYRAX_ENGINE_PATH && \
-  bundle install --jobs "$(nproc)" && \
-  yarn && yarn cache clean
-
-RUN RAILS_ENV=production SECRET_KEY_BASE='fakesecret1234' DB_ADAPTER=nulldb DATABASE_URL='postgresql://fake' bundle exec rake assets:precompile
-
-
-FROM hyrax-worker-base as hyrax-engine-dev-worker
-
-ARG APP_PATH=.dassie
-ARG BUNDLE_WITHOUT=
-
-ENV HYRAX_ENGINE_PATH /app/samvera/hyrax-engine
-
-COPY --chown=1001:101 $APP_PATH /app/samvera/hyrax-webapp
-COPY --chown=1001:101 . /app/samvera/hyrax-engine
+  BUNDLE_GEMFILE=Gemfile.dassie bundle install --jobs "$(nproc)" && yarn && \
+  cd $HYRAX_ENGINE_PATH && bundle install --jobs "$(nproc)" && yarn && \
+  yarn cache clean
 
-RUN bundle install --jobs "$(nproc)"
+ENTRYPOINT ["dev-entrypoint.sh"]
+CMD ["bundle", "exec", "puma", "-v", "-b", "tcp://0.0.0.0:3000"]