-
Notifications
You must be signed in to change notification settings - Fork 130
feat: Add PostgreSQL database backend support #439
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 12 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
4186c91
feat: Add dual database support (SQLite + Postgres)
phernandez e258221
add postgres wip
phernandez 26f51d0
Merge branch 'main' into feature/postgres-support
phernandez 150fc54
feat: Add dual database backend test infrastructure and improve test β¦
phernandez 00cd417
sqlite tests pass, postgres fails PRAGMA error
phernandez cc34784
unit tests passing
phernandez 7a91a7b
sqlite and postgres tests passing
phernandez 891d311
feat: Add PostgreSQL database backend support with dual-backend archiβ¦
phernandez 9d99436
Merge main branch with GitHub Actions Postgres testing updates
phernandez ae97f19
ci: Optimize test execution by splitting SQLite and Postgres tests
phernandez 7b9c438
fix test fixtures for test_project_commands_integration
phernandez b186518
ci: Split test jobs to fix Windows container error
phernandez 0826fa1
test: Skip non-Windows pool test on Windows
phernandez File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # Basic Memory Environment Variables Example | ||
| # Copy this file to .env and customize as needed | ||
| # Note: .env files are gitignored and should never be committed | ||
|
|
||
| # ============================================================================ | ||
| # PostgreSQL Test Database Configuration | ||
| # ============================================================================ | ||
| # These variables allow you to override the default test database credentials | ||
| # Default values match docker-compose-postgres.yml for local development | ||
| # | ||
| # Only needed if you want to use different credentials or a remote test database | ||
| # By default, tests use: postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test | ||
|
|
||
| # Full PostgreSQL test database URL (used by tests and migrations) | ||
| # POSTGRES_TEST_URL=postgresql+asyncpg://basic_memory_user:dev_password@localhost:5433/basic_memory_test | ||
|
|
||
| # Individual components (used by justfile postgres-reset command) | ||
| # POSTGRES_USER=basic_memory_user | ||
| # POSTGRES_TEST_DB=basic_memory_test | ||
|
|
||
| # ============================================================================ | ||
| # Production Database Configuration | ||
| # ============================================================================ | ||
| # For production use, set these in your deployment environment | ||
| # DO NOT use the test credentials above in production! | ||
|
|
||
| # BASIC_MEMORY_DATABASE_BACKEND=postgres # or "sqlite" | ||
| # BASIC_MEMORY_DATABASE_URL=postgresql+asyncpg://user:password@host:port/database |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -433,6 +433,57 @@ See the [Documentation](https://memory.basicmachines.co/) for more info, includi | |
| - [Managing multiple Projects](https://docs.basicmemory.com/guides/cli-reference/#project) | ||
| - [Importing data from OpenAI/Claude Projects](https://docs.basicmemory.com/guides/cli-reference/#import) | ||
|
|
||
| ## Development | ||
|
|
||
| ### Running Tests | ||
|
|
||
| Basic Memory supports dual database backends (SQLite and Postgres). Tests are parametrized to run against both backends automatically. | ||
|
|
||
| **Quick Start:** | ||
| ```bash | ||
| # Run SQLite tests (default, no Docker needed) | ||
| just test-sqlite | ||
|
|
||
| # Run Postgres tests (requires Docker) | ||
| just test-postgres | ||
| ``` | ||
|
|
||
| **Available Test Commands:** | ||
|
|
||
| - `just test-sqlite` - Run tests against SQLite only (fastest, no Docker needed) | ||
| - `just test-postgres` - Run tests against Postgres only (requires Docker) | ||
| - `just test-windows` - Run Windows-specific tests (auto-skips on other platforms) | ||
| - `just test-benchmark` - Run performance benchmark tests | ||
| - `just test-all` - Run all tests including Windows, Postgres, and benchmarks | ||
|
|
||
| **Postgres Testing Requirements:** | ||
|
|
||
| To run Postgres tests, you need to start the test database: | ||
| ```bash | ||
| docker-compose -f docker-compose-postgres.yml up -d | ||
| ``` | ||
|
|
||
| Tests will connect to `localhost:5433/basic_memory_test`. | ||
|
|
||
| **Test Markers:** | ||
|
|
||
| Tests use pytest markers for selective execution: | ||
| - `postgres` - Tests that run against Postgres backend | ||
| - `windows` - Windows-specific database optimizations | ||
| - `benchmark` - Performance tests (excluded from default runs) | ||
|
|
||
| **Other Development Commands:** | ||
| ```bash | ||
| just install # Install with dev dependencies | ||
| just lint # Run linting checks | ||
| just typecheck # Run type checking | ||
| just format # Format code with ruff | ||
| just check # Run all quality checks | ||
| just migration "msg" # Create database migration | ||
| ``` | ||
|
|
||
| See the [justfile](justfile) for the complete list of development commands. | ||
|
|
||
| ## License | ||
|
|
||
| AGPL-3.0 | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Docker Compose configuration for Basic Memory with PostgreSQL | ||
| # Use this for local development and testing with Postgres backend | ||
| # | ||
| # Usage: | ||
| # docker-compose -f docker-compose-postgres.yml up -d | ||
| # docker-compose -f docker-compose-postgres.yml down | ||
|
|
||
| services: | ||
| postgres: | ||
| image: postgres:17 | ||
| container_name: basic-memory-postgres | ||
| environment: | ||
| # Local development/test credentials - NOT for production | ||
| # These values are referenced by tests and justfile commands | ||
| POSTGRES_DB: basic_memory | ||
| POSTGRES_USER: basic_memory_user | ||
| POSTGRES_PASSWORD: dev_password # Simple password for local testing only | ||
| ports: | ||
| - "5433:5432" | ||
| volumes: | ||
| - postgres_data:/var/lib/postgresql/data | ||
| healthcheck: | ||
| test: ["CMD-SHELL", "pg_isready -U basic_memory_user -d basic_memory"] | ||
| interval: 10s | ||
| timeout: 5s | ||
| retries: 5 | ||
| restart: unless-stopped | ||
|
|
||
| volumes: | ||
| # Named volume for Postgres data | ||
| postgres_data: | ||
| driver: local | ||
|
|
||
| # Named volume for persistent configuration | ||
| # Database will be stored in Postgres, not in this volume | ||
| basic-memory-config: | ||
| driver: local | ||
|
|
||
| # Network configuration (optional) | ||
| # networks: | ||
| # basic-memory-net: | ||
| # driver: bridge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,16 +7,78 @@ install: | |
| @echo "" | ||
| @echo "π‘ Remember to activate the virtual environment by running: source .venv/bin/activate" | ||
|
|
||
| # Run all tests with unified coverage report | ||
| test: test-unit test-int | ||
|
|
||
| # Run unit tests only (fast, no coverage) | ||
| test-unit: | ||
| uv run pytest -p pytest_mock -v --no-cov -n auto tests | ||
| uv run pytest -p pytest_mock -v --no-cov tests | ||
|
|
||
| # Run integration tests only (fast, no coverage) | ||
| test-int: | ||
| uv run pytest -p pytest_mock -v --no-cov -n auto test-int | ||
|
|
||
| # Run all tests with unified coverage report | ||
| test: test-unit test-int | ||
| uv run pytest -p pytest_mock -v --no-cov test-int | ||
|
|
||
| # ============================================================================== | ||
| # DATABASE BACKEND TESTING | ||
| # ============================================================================== | ||
| # Basic Memory supports dual database backends (SQLite and Postgres). | ||
| # Tests are parametrized to run against both backends automatically. | ||
| # | ||
| # Quick Start: | ||
| # just test-sqlite # Run SQLite tests (default, no Docker needed) | ||
| # just test-postgres # Run Postgres tests (requires Docker) | ||
| # | ||
| # For Postgres tests, first start the database: | ||
| # docker-compose -f docker-compose-postgres.yml up -d | ||
| # ============================================================================== | ||
|
|
||
| # Run tests against SQLite only (default backend, skip Postgres/Benchmark tests) | ||
| # This is the fastest option and doesn't require any Docker setup. | ||
| # Use this for local development and quick feedback. | ||
| # Includes Windows-specific tests which will auto-skip on non-Windows platforms. | ||
| test-sqlite: | ||
| uv run pytest -p pytest_mock -v --no-cov -m "not postgres and not benchmark" tests test-int | ||
|
|
||
| # Run tests against Postgres only (requires docker-compose-postgres.yml up) | ||
| # First start Postgres: docker-compose -f docker-compose-postgres.yml up -d | ||
| # Tests will connect to localhost:5433/basic_memory_test | ||
| # To reset the database: just postgres-reset | ||
| test-postgres: | ||
| uv run pytest -p pytest_mock -v --no-cov -m "postgres and not benchmark" tests test-int | ||
|
|
||
| # Reset Postgres test database (drops and recreates schema) | ||
| # Useful when Alembic migration state gets out of sync during development | ||
| # Uses credentials from docker-compose-postgres.yml | ||
| postgres-reset: | ||
| docker exec basic-memory-postgres psql -U ${POSTGRES_USER:-basic_memory_user} -d ${POSTGRES_TEST_DB:-basic_memory_test} -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;" | ||
| @echo "β Postgres test database reset" | ||
|
|
||
| # Run Alembic migrations manually against Postgres test database | ||
| # Useful for debugging migration issues | ||
| # Uses credentials from docker-compose-postgres.yml (can override with env vars) | ||
| postgres-migrate: | ||
| @cd src/basic_memory/alembic && \ | ||
| BASIC_MEMORY_DATABASE_BACKEND=postgres \ | ||
| BASIC_MEMORY_DATABASE_URL=${POSTGRES_TEST_URL:-postgresql://basic_memory_user:dev_password@localhost:5433/basic_memory_test} \ | ||
| uv run alembic upgrade head | ||
| @echo "β Migrations applied to Postgres test database" | ||
|
|
||
| # Run Windows-specific tests only (only works on Windows platform) | ||
| # These tests verify Windows-specific database optimizations (locking mode, NullPool) | ||
| # Will be skipped automatically on non-Windows platforms | ||
| test-windows: | ||
| uv run pytest -p pytest_mock -v --no-cov -m windows tests test-int | ||
|
|
||
| # Run benchmark tests only (performance testing) | ||
| # These are slow tests that measure sync performance with various file counts | ||
| # Excluded from default test runs to keep CI fast | ||
| test-benchmark: | ||
| uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int | ||
|
|
||
| # Run all tests including Windows, Postgres, and Benchmarks (for CI/comprehensive testing) | ||
| # Use this before releasing to ensure everything works across all backends and platforms | ||
| test-all: | ||
| uv run pytest -p pytest_mock -v --no-cov tests test-int | ||
|
|
||
| # Generate HTML coverage report | ||
| coverage: | ||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium test
Copilot Autofix
AI 3 days ago
The best way to fix this problem is to explicitly declare a
permissionsblock at the root level of the workflow (line 2, immediately aftername:), applying to all jobs by default. For most test workflows that do not require writing to the repo or interacting with PRs, the minimal permissions arecontents: read. If some jobs require greater access, those jobs can declare their ownpermissionsblock as necessary. In this case, none of the shown jobs appear to require write access, so addingimmediately after the
name: Testsline is the simplest fix. No additional imports or package installations are needed.