Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: CI

on:
push:
branches: [main, develop]
pull_request:
branches: [main, develop]

jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- run: npm run lint

test:
name: Test
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['20', '22']
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
- run: npm ci
- run: npm test

format-check:
name: Format Check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- run: npm run format:check
2 changes: 2 additions & 0 deletions .husky/pre-commit
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
npm run lint
npm test
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added

- Rolling window mode for automatic data window maintenance
- `clear` command to truncate table without deleting schema
- Automatic backfill on service start
Expand All @@ -26,19 +27,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Progress tracking for all operations

### Changed

- `start` command now auto-backfills by default (rolling window mode)
- Documentation reorganized into logical sections
- Test files moved to examples/ directory
- Improved error messages and user feedback

### Fixed

- Configuration loading from config.yaml
- BigQuery credential handling
- Service account key path resolution

## [1.0.0] - 2024-XX-XX

### Added

- Initial release of BigQuery Plugin for HarperDB
- Modulo-based partitioning for distributed ingestion
- Adaptive batch sizing based on sync lag
Expand All @@ -51,6 +55,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Configurable sync phases (initial, catchup, steady)

### Plugin Features

- Horizontal scalability with linear throughput
- No coordination overhead between nodes
- Deterministic partition assignments
Expand All @@ -60,6 +65,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Generic storage for any BigQuery schema

### Documentation

- Comprehensive design document
- Blog post explaining architecture evolution
- System overview showing component interaction
Expand All @@ -86,6 +92,7 @@ This project follows [Semantic Versioning](https://semver.org/):
### Deprecation Policy

Features will be deprecated with:

1. Warning in release notes
2. Deprecation notice in code
3. Minimum 2 minor versions before removal
Expand Down
6 changes: 6 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ git commit -m "chore: update dependencies"
```

**Commit Types**:

- `feat`: New feature
- `fix`: Bug fix
- `docs`: Documentation changes
Expand All @@ -86,6 +87,7 @@ git push origin your-branch-name
```

Then create a Pull Request on GitHub with:

- Clear title and description
- Reference any related issues
- Screenshots/examples if applicable
Expand All @@ -101,6 +103,7 @@ Then create a Pull Request on GitHub with:
- Keep functions focused and small

**Example**:

```javascript
/**
* Generate vessel position data
Expand Down Expand Up @@ -190,6 +193,7 @@ The BigQuery plugin integrates with HarperDB. When modifying plugin code:
4. Check data consistency

**Key Files**:

- `src/sync-engine.js` - Main sync engine logic
- `src/validation.js` - Data validation
- `schema/harper-bigquery-sync.graphql` - GraphQL schema
Expand All @@ -204,6 +208,7 @@ The maritime data synthesizer generates test data. When modifying:
4. Check rolling window behavior

**Key Files**:

- `src/generator.js` - Data generation
- `src/service.js` - Orchestration
- `src/bigquery.js` - BigQuery client
Expand Down Expand Up @@ -233,6 +238,7 @@ By contributing, you agree that your contributions will be licensed under the Ap
## Recognition

Contributors will be:

- Listed in package.json contributors
- Acknowledged in release notes
- Credited in documentation for major features
Expand Down
Loading