Persistent Installation, Configuration, and Build Failures with Aura

[packetlost]

Description
I have encountered persistent installation and configuration issues with the Aura repository, both during the initial setup and following multiple attempts to completely reinstall everything from scratch. Below is a detailed description of the errors, actions taken, and the difficulties encountered when attempting to resolve them.

System Details
OS: Ubuntu 24.04.1 LTS running on WSL2
Docker Version: 26.1.3
Docker Compose Version: 1.29.2
Aura GitHub Repo: Aura
Node.js Version: v20.18.2
Rust Version: 1.84 (used for building Aura dependencies)

Initial Errors During First Installation

Docker Compose Configuration Issues:

ERROR: The Compose file './docker-compose.yaml' is invalid because:
services.backfill.restart contains an invalid type, it should be a string
services.integrity-verification.volumes value ['./creds.json:/usr/src/app/creds.json', '::rw', '::rw'] has non-unique elements
services.das-api.volumes value ['::ro', '/api::rw', '::ro', '::rw', './heaps:/usr/src/app/heaps:rw'] has non-unique elements
services.rocksdb-backup.volumes value ['::ro', '::rw', '::rw', '/backup::rw'] has non-unique elements
services.ingester.volumes value ['::rw', '/tmp/rocks_dump:/tmp/rocks_dump:rw', '::rw', '::rw', '::rw', '::rw', '::rw', '::ro', '::rw', './creds.json:/usr/src/app/creds.json', './migrations:/usr/src/app/migrations', './arweave_wallet.json:/usr/src/app/arweave_wallet.json', './heaps:/usr/src/app/heaps:rw'] has non-unique elements

Build Errors with Rust Cargo:

[planner 15/15] RUN cargo chef prepare --recipe-path recipe.json:
Error: Failed to compute recipe

Caused by:
0: Cannot extract Cargo metadata
1: cargo metadata exited with an error: error: failed to load manifest for workspace member /rust/consistency_check
referenced by workspace at /rust/Cargo.toml

Caused by:
failed to read /rust/consistency_check/Cargo.toml

Caused by:
No such file or directory (os error 2)

Invalid Docker Compose Volumes and Missing Variables:

WARNING: The ROCKS_DB_PATH variable is not set. Defaulting to a blank string.
WARNING: The FILE_STORAGE_PATH variable is not set. Defaulting to a blank string.
...
ERROR: The Compose file './docker-compose.yaml' is invalid because:
services.ingester.volumes value ['::rw', '/tmp/rocks_dump:/tmp/rocks_dump:rw', '::rw', '::rw', './creds.json:/usr/src/app/creds.json'] has non-unique elements
Actions Attempted
Reinstalled Docker, Docker Compose, and Aura:

Fully removed Docker, Docker Compose, and all Aura-related directories.

Ran the following commands:

sudo apt remove docker docker-compose
sudo apt purge docker docker-compose
sudo rm -rf ~/.docker
docker system prune -a --volumes -f
rm -rf ~/deepaura/aura
git clone https://github.com/metaplex-foundation/aura.git ~/deepaura/aura

Reset .env File:

Updated .env with valid paths for variables such as ROCKS_DB_PATH, ROCKS_DB_PATH_CONTAINER, and FILE_STORAGE_PATH.

Modified docker-compose.yaml:

Manually removed duplicate or non-unique elements in the volumes and invalid restart types in the configuration file.
These edits temporarily resolved certain warnings, but the build process failed consistently.

Blockbuster Issue:

Blockbuster Directory Missing

When attempting to build the integrity-verification service, the following error occurs:

[planner 15/16] COPY blockbuster ./blockbuster

integrity_verification.Dockerfile:21

19: COPY usecase ./usecase
20: COPY integrity_verification ./integrity_verification
21: COPY blockbuster ./blockbuster

ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref b815666d-1d86-4fdf-9d82-54fc5877026d::3sra4vmtwnlqal0slgysrqilv: "/blockbuster": not found
ERROR: Service 'integrity-verification' failed to build : Build failed

Observations:

The integrity_verification.Dockerfile includes a line to copy the blockbuster directory, but this directory does not exist in the repository.
This breaks the build process as it cannot proceed without the required files.
Steps to Reproduce:

Clone the repository.
Run docker-compose up --build following the documented steps.
Observe the failure due to the missing blockbuster directory.

Errors Encountered After Reinstallation

Even after performing a complete reinstall of all dependencies and the Aura repository, the same errors resurfaced. Specifically:

Docker Compose Configuration:

ERROR: The Compose file './docker-compose.yaml' is invalid because:
services.backfill.restart contains an invalid type, it should be a string
services.integrity-verification.volumes value ['./creds.json:/usr/src/app/creds.json', '::rw', '::rw'] has non-unique elements

Rust Build Failure:

[planner 15/15] RUN cargo chef prepare --recipe-path recipe.json:
Error: Failed to compute recipe

Caused by:
0: Cannot extract Cargo metadata
1: cargo metadata exited with an error: error: failed to load manifest for workspace member /rust/consistency_check

Missing Dependencies:

WARNING: The ROCKS_DB_PATH variable is not set. Defaulting to a blank string.
WARNING: The FILE_STORAGE_PATH variable is not set. Defaulting to a blank string.

Questions and Request for Support

Why does the Aura repository include invalid docker-compose.yaml configurations by default? Should this file be updated in the repository?

How can the cargo chef prepare step be fixed? It consistently fails due to missing Rust workspace members (e.g., /rust/consistency_check).

Is there a documented list of environment variables required for a successful installation? Many variables like ROCKS_DB_PATH and FILE_STORAGE_PATH seem necessary but are not clearly documented.

Could there be compatibility issues with Ubuntu 24.04, Docker Compose v1.29.2, or Rust 1.84?

I do not know how to proceed...

[armyhaylenko]

hey, we're actively working on improving the build and the infrastructure overall. if you take a look at the latest develop branch, you can see updates to dockerfiles and ci.
as for the docker-compose, you need to provide the env before you actually try and launch docker-compose. i personally like to do it the following way:

eval $(cat .env) docker compose up -d <service>

there are also more PRs on their way that simplify the required env and deduplicate the environment variables needed for compose launch. we really appreciate you trying to operate Aura, and we will do our best to fix most of these issues in the following weeks.

TLDR; please try the latest develop & let us know if you have more specific questions afterwards.