Skip to content
Closed
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
3bda78f
Add support for incrementing version number
na-ka-na May 13, 2025
011e3a3
Update README
na-ka-na Jun 11, 2025
9b88963
feat: WIP
mscottx88 Jun 18, 2025
77237a5
feat: WIP
Jun 24, 2025
710731a
feat: WIP
Jun 24, 2025
aa2a651
feat: WIP
Jun 24, 2025
83b9aea
feat: wip
Jun 24, 2025
627e1c4
feat: fixes
Jun 26, 2025
e5d8e41
Update versioning section in the README with actual subscriptions tab…
na-ka-na Jun 26, 2025
da0dafa
feat: tests overhaul WIP
Jun 27, 2025
1705e31
feat: formatting
Jun 27, 2025
43ebcd9
feat: WIP
Jun 27, 2025
a5b09f4
feat: wip
Jun 27, 2025
06956b4
feat: WIP
Jun 27, 2025
9225573
feat: wip
Jun 27, 2025
195e63b
feat: actions
Jun 27, 2025
99d7b57
feat: fix test runner
Jun 30, 2025
5eecb59
feat: fix scripts
Jun 30, 2025
0611a2b
feat: keep it DRY
Jun 30, 2025
db684ee
feat: server versions
Jun 30, 2025
9fe51d4
feat: server versions
Jun 30, 2025
ffc0548
Add tests for increment version
na-ka-na Jun 30, 2025
b432ba8
feat: wip
Jun 30, 2025
c5d844e
feat: wip
Jun 30, 2025
3ff101b
feat: wip
Jun 30, 2025
7ff495e
feat: docs
Jun 30, 2025
fe3f532
Merge commit '08fff5de0f2f2618917f00e05b105dd0ffe5d6d9' into feature/…
Jul 1, 2025
81d95bd
feat: docs
Jul 1, 2025
452d7ed
feat: lint
Jul 1, 2025
03631bc
feat: comparison reports
Jul 2, 2025
b3e3300
feat: wip
Jul 2, 2025
0a880c1
feat: tests
Jul 2, 2025
6cd52ce
Merge branch 'master' of https://github.com/na-ka-na/temporal_tables
Jul 2, 2025
6eda315
chore: revert
Jul 2, 2025
e659a8a
chore: revert
Jul 2, 2025
b085f74
chore: cleanup
Jul 2, 2025
73f990a
feat: docs
Jul 2, 2025
f8b2163
chore: revert
Jul 2, 2025
3152d31
Merge branch 'master' of https://github.com/nearform/temporal_tables
Jul 10, 2025
3786584
Merge branch 'master' into feature/static-trigger-generator
Jul 10, 2025
bd346ae
feat: wip
Jul 11, 2025
489ff3c
feat: wip
Jul 11, 2025
f8e0a22
feat: wip
Jul 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .commitlintrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"extends": [
"@commitlint/config-conventional"
]
}
6 changes: 6 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[*]
end_of_line = lf
insert_final_newline = true
charset = utf-8
indent_style = space
indent_size = 2
8 changes: 8 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,11 @@ jobs:
- name: Run tests no check
run: |
make run_test_nochecks

- name: Run Node.js e2e Tests
uses: actions/setup-node@v4
with:
node-version: '20.x'
- run: npm ci
- run: npm run build --if-present
- run: npm run test:e2e:runner
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
test/result
.env
.envrc
.eslintcache
test/remote_expected
test/remote_sql
test/remote_result
node_modules
1 change: 1 addition & 0 deletions .nvmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
lts/*
6 changes: 6 additions & 0 deletions .prettierrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"semi": false,
"singleQuote": true,
"arrowParens": "avoid",
"trailingComma": "none"
}
24 changes: 24 additions & 0 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"version": "0.2.0",
"configurations": [
{
"type": "node",
"request": "launch",
"name": "Test File w/ ts-node",
"protocol": "inspector",
"runtimeArgs": ["--loader", "ts-node/esm/transpile-only", "--test"],
"args": ["${relativeFile}"],
"outputCapture": "std",
"internalConsoleOptions": "openOnSessionStart",
"envFile": "${workspaceRoot}/.env",
"skipFiles": [
"${workspaceRoot}/../../node_modules/**/*",
"<node_internals>/**/*"
],
"windows": {
"skipFiles": ["C:\\**\\node_modules\\**\\*", "<node_internals>/**/*"]
},
"disableOptimisticBPs": true
}
]
}
285 changes: 283 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,19 @@ Over time, new features have been introduced while maintaining backward compatib

- [Ignore updates with no actual changes](#ignore-unchanged-values)
- [Include the current version in history](#include-current-version-in-history)
- [Static trigger code generation](#static-trigger-code-generation)
- [Automatic trigger re-rendering](#event-trigger-for-automatic-re-rendering)
- [Versioning tables metadata management](#versioning-tables-metadata)
- [Update conflict mitigation](#mitigate-update-conflicts)
- [Migration mode support](#migration-mode)

<a name="usage"></a>

## PostgreSQL Version Requirements

- **Legacy functionality** (basic versioning): Works with any PostgreSQL version
- **Modern functionality** (static triggers, metadata management): Requires PostgreSQL 13.21 or higher

## Usage

Create a database and the versioning function:
Expand Down Expand Up @@ -234,6 +244,40 @@ FOR EACH ROW EXECUTE PROCEDURE versioning(
);
```

<a name="mitigate-update-conflicts"></a>

### Mitigate Update Conflicts

By default, when multiple transactions try to update the same row simultaneously, PostgreSQL's versioning system can detect conflicts where the system period start time would be greater than or equal to the current transaction time. This can cause errors in high-concurrency scenarios.

The `mitigate_update_conflicts` parameter (sixth parameter) automatically adjusts timestamps by adding 1 microsecond when conflicts are detected, allowing operations to proceed smoothly:

```sql
CREATE TRIGGER versioning_trigger
BEFORE INSERT OR UPDATE OR DELETE ON subscriptions
FOR EACH ROW EXECUTE PROCEDURE versioning(
'sys_period', 'subscriptions_history', true, false, false, true
);
```

**Note:** This feature slightly modifies timestamps to resolve conflicts, which may affect temporal queries that rely on exact timing.

<a name="migration-mode"></a>

### Migration Mode

Migration mode (seventh parameter) enables gradual adoption of the `include_current_version_in_history` feature for existing tables without requiring a maintenance window:

```sql
CREATE TRIGGER versioning_trigger
BEFORE INSERT OR UPDATE OR DELETE ON subscriptions
FOR EACH ROW EXECUTE PROCEDURE versioning(
'sys_period', 'subscriptions_history', true, false, true, false, true
);
```

When enabled, the trigger automatically populates missing current versions in the history table during UPDATE or DELETE operations.

<a name="migration-to-include-current-version-in-history"></a>

### Migrating to include_current_version_in_history
Expand Down Expand Up @@ -339,13 +383,13 @@ WHERE UPPER(sys_period) IS NULL;

When adopting the `include_current_version_in_history` feature for existing tables, you can use the automatic gradual migration mode to seamlessly populate the history table with current records.

The migration mode is enabled by adding a sixth parameter to the versioning trigger:
The migration mode is enabled by adding a seventh parameter to the versioning trigger:

```sql
CREATE TRIGGER versioning_trigger
BEFORE INSERT OR UPDATE OR DELETE ON your_table
FOR EACH ROW EXECUTE PROCEDURE versioning(
'sys_period', 'your_table_history', true, false, true, true
'sys_period', 'your_table_history', true, false, true, false, true
);
```

Expand Down Expand Up @@ -385,6 +429,174 @@ When migration mode is enabled:

**Note:** The automatic migration happens gradually, filling in missing history only when existing records are updated or deleted. As a result, records that rarely change will still require manual migration using the [method described above](#migration-to-include-current-version-in-history). However, since the most active records will be automatically migrated, the risk of missing important data is greatly reduced, eliminating the need for a dedicated maintenance window.

<a name="versioning-tables-metadata"></a>

## Versioning Tables Metadata

The modern functionality includes a metadata table to track all versioned tables and their configuration. This enables automatic trigger re-rendering when table schemas change.

### Setup

1. **Install the metadata table:**
```sh
psql temporal_test < versioning_tables_metadata.sql
```

2. **Register versioned tables:**
```sql
INSERT INTO versioning_tables_metadata (
table_name,
table_schema,
history_table,
history_table_schema,
sys_period,
ignore_unchanged_values,
include_current_version_in_history,
mitigate_update_conflicts,
enable_migration_mode
) VALUES (
'subscriptions',
'public',
'subscriptions_history',
'public',
'sys_period',
false,
false,
false,
false
);
```

### Benefits

- **Automatic trigger re-rendering** when table schemas change
- **Centralized configuration** for all versioned tables
- **Audit trail** with created_at and updated_at timestamps
- **Schema flexibility** with separate schemas for tables and history tables

<a name="static-trigger-code-generation"></a>

## Static Trigger Code Generation (PostgreSQL 13+)

The modern static trigger generator creates optimized, table-specific trigger functions with no runtime schema lookups, providing better performance and reliability.

### Basic Usage

1. **Install the generator:**
```sh
psql temporal_test < generate_static_versioning_trigger.sql
psql temporal_test < render_versioning_trigger.sql
```

2. **Generate static trigger:**
```sql
-- Using the generator function directly
SELECT generate_static_versioning_trigger(
p_table_name => 'subscriptions',
p_history_table => 'subscriptions_history',
p_sys_period => 'sys_period',
p_ignore_unchanged_values => false,
p_include_current_version_in_history => false,
p_mitigate_update_conflicts => false,
p_enable_migration_mode => false
);
```

3. **Using the render procedure (recommended):**
```sql
CALL render_versioning_trigger(
p_table_name => 'subscriptions',
p_history_table => 'subscriptions_history',
p_sys_period => 'sys_period',
p_ignore_unchanged_values => false,
p_include_current_version_in_history => false,
p_mitigate_update_conflicts => false,
p_enable_migration_mode => false
);
```

### Advanced Features

The static generator supports all modern features:

```sql
-- Generate trigger with all advanced features enabled
CALL render_versioning_trigger(
p_table_name => 'subscriptions',
p_history_table => 'subscriptions_history',
p_sys_period => 'sys_period',
p_ignore_unchanged_values => true,
p_include_current_version_in_history => true,
p_mitigate_update_conflicts => true,
p_enable_migration_mode => true
);
```

### Benefits of Static Triggers

- **Better Performance**: No runtime schema lookups
- **Compile-time Validation**: Errors detected when trigger is created
- **Explicit Dependencies**: Clear relationship between table structure and trigger code
- **Optimized Code**: Generated specifically for your table's current schema

<a name="event-trigger-for-automatic-re-rendering"></a>

## Event Trigger for Automatic Re-rendering (PostgreSQL 13+)

Event triggers automatically re-render static versioning triggers when table schemas change, ensuring triggers stay synchronized with table structures.

### Setup

1. **Install event trigger:**
```sh
psql temporal_test < event_trigger_versioning.sql
```

2. **Ensure metadata is populated:**
The event trigger uses the `versioning_tables_metadata` table to determine which tables need trigger re-rendering.

### How It Works

- **Automatic Detection**: Event trigger fires on `ALTER TABLE` commands
- **Metadata Lookup**: Checks if the altered table is registered in `versioning_tables_metadata`
- **Trigger Re-rendering**: Automatically calls `render_versioning_trigger` with stored configuration
- **Schema Synchronization**: Ensures triggers always match current table structure

### Example Workflow

```sql
-- 1. Register table in metadata
INSERT INTO versioning_tables_metadata (
table_name, table_schema, history_table, history_table_schema,
sys_period, ignore_unchanged_values, include_current_version_in_history
) VALUES (
'subscriptions', 'public', 'subscriptions_history', 'public',
'sys_period', true, true
);

-- 2. Generate initial trigger
CALL render_versioning_trigger(
p_table_name => 'subscriptions',
p_history_table => 'subscriptions_history',
p_sys_period => 'sys_period',
p_ignore_unchanged_values => true,
p_include_current_version_in_history => true
);

-- 3. Modify table schema - trigger is automatically re-rendered
ALTER TABLE subscriptions ADD COLUMN plan text;
ALTER TABLE subscriptions_history ADD COLUMN plan text;

-- The event trigger automatically regenerates the versioning trigger!
```

### Benefits

- **Zero Maintenance**: Triggers stay synchronized automatically
- **Schema Evolution**: Safe to modify table structures
- **Error Prevention**: Eliminates manual trigger update requirements
- **Development Friendly**: Works seamlessly with migration scripts

<a name="migrations"></a>

## Migrations
Expand All @@ -411,6 +623,75 @@ If the column doesn't accept null values you'll need to modify it to allow for n

## Test

### End-to-End Tests (Enhanced)

We've enhanced our comprehensive end-to-end tests written in modern TypeScript using Node.js built-in test runner and node-postgres. These tests provide extensive coverage of all temporal table features with improved reliability:

#### Test Coverage
- **Static Generator Tests**: Full coverage of the static trigger generator functionality
- **Legacy Function Tests**: Backward compatibility testing with the original versioning function
- **Event Trigger Tests**: Automatic trigger re-rendering on schema changes
- **Integration Tests**: Real-world scenarios including e-commerce workflows, schema evolution, and performance testing
- **Error Handling**: Edge cases, transaction rollbacks, and concurrent modifications
- **Version-aware Testing**: Automatically adapts tests based on PostgreSQL version

#### Test Reliability Improvements
- **Enhanced timestamp handling**: Eliminates flaky tests due to timing issues
- **Robust database state management**: Proper cleanup and isolation between tests
- **Version-specific loading**: Only loads SQL files appropriate for the PostgreSQL version
- **Better error reporting**: Detailed failure messages with timing context

#### Running E2E Tests

1. **Prerequisites**: Ensure you have PostgreSQL running (Docker or local installation)

2. **Install dependencies**:
```bash
npm install
```

3. **Start database** (if using Docker):
```bash
npm run db:start
```

4. **Run all E2E tests**:
```bash
npm run test:e2e
```

5. **Run specific test suites**:
```bash
# Static generator tests
npm run test:e2e:static

# Legacy function tests
npm run test:e2e:legacy

# Integration tests
npm run test:e2e:integration

# Event trigger tests
npm run test:e2e:event

# Custom test runner (TypeScript)
npm run test:e2e:runner
```

#### Features of Enhanced E2E Tests
- **Type-safe**: Written in TypeScript with proper type definitions
- **Modern**: Uses Node.js built-in test runner (no external dependencies like Jest)
- **Comprehensive**: Tests all features including advanced options and edge cases
- **Isolated**: Each test starts with a clean database state
- **Real-world scenarios**: Includes practical examples like e-commerce order processing
- **Performance testing**: Bulk operations and concurrent access patterns
- **Version-aware**: Automatically skips tests not supported by current PostgreSQL version
- **Reliable timing**: Enhanced timestamp checking eliminates flaky test failures

See [test/e2e/README.md](test/e2e/README.md) for detailed documentation.

### Traditional Tests

Ensure you have a postgres database available. A database container can be started by running:

```sh
Expand Down
Loading