Skip to content

Conversation

@yokofly
Copy link
Collaborator

@yokofly yokofly commented Jan 7, 2026

Please write user-readable short description of the changes:
fix #1079

  -- Source streams
  DROP STREAM IF EXISTS orders;
  DROP STREAM IF EXISTS products;

  CREATE STREAM orders(order_id int32, product_id int32, quantity int32);
  CREATE STREAM products(product_id int32, name string)
      PRIMARY KEY product_id SETTINGS mode='versioned_kv';

  -- Target stream
  DROP STREAM IF EXISTS enriched_orders;
  CREATE STREAM enriched_orders(order_id int32, product_name string, quantity int32);

  -- MV with join using hybrid hash join and incremental checkpoint
  DROP VIEW IF EXISTS mv_join_incr_ckpt;

  CREATE MATERIALIZED VIEW mv_join_incr_ckpt
  INTO enriched_orders
  AS
  SELECT
      o.order_id,
      p.name AS product_name,
      o.quantity
  FROM orders AS o
  INNER JOIN products AS p ON o.product_id = p.product_id
  SETTINGS
      seek_to = 'earliest',
      checkpoint_interval = 2,
      default_hash_join = 'hybrid',
      checkpoint_settings = 'type=auto;incremental=true';

  -- Insert product data
  INSERT INTO products(product_id, name) VALUES (1, 'Widget'), (2, 'Gadget');

  -- Insert orders
  INSERT INTO orders(order_id, product_id, quantity) VALUES (100, 1, 5), (101, 2, 3);

  -- Wait for epoch 1 checkpoint
  SELECT sleep(3) FORMAT Null;

  -- Insert more data to trigger activity for epoch 2
  INSERT INTO orders(order_id, product_id, quantity) VALUES (102, 1, 10);

  -- Wait for epoch 2 checkpoint (incremental should kick in here)
  SELECT sleep(3) FORMAT Null;

  -- Insert more data for epoch 3
  INSERT INTO orders(order_id, product_id, quantity) VALUES (103, 2, 7);

  -- Wait for epoch 3 checkpoint
  SELECT sleep(3) FORMAT Null;


old:

2026.01.07 04:44:27.708451 [ 220 ] {.mv-047c9b6a-1fa9-4ac1-8e32-0dc7a987929b} <Information> RocksCheckpoint: Took 13 ms to materialized rocks ckpt to 
.mv-047c9b6a-1fa9-4ac1-8e32-0dc7a987929b/3_01KEBCAFJQN2GNX0430PCBDPM2/1.rckpt on disk local_fs_ckpt_storage (/var/lib/proton/checkpoint/), materialized_bytes=2
2026.01.07 04:44:27.708507 [ 220 ] {.mv-047c9b6a-1fa9-4ac1-8e32-0dc7a987929b} <Information> LocalFileSystemCheckpointStorage: Committed checkpoint 
epoch=3_01KEBCAFJQN2GNX0430PCBDPM2 for query=.mv-047c9b6a-1fa9-4ac1-8e32-0dc7a987929b

new:

2026.01.07 04:42:04.211291 [ 5826 ] {.mv-8e9f4216-34cc-4547-b601-fcf461ffb70e} <Information> RocksCheckpoint: Took 12 ms to materialized incremental rocks ckpt to 
.mv-8e9f4216-34cc-4547-b601-fcf461ffb70e/3_01KEBC63H429WKM6YEXZSM1Z09/1.rckpt on disk local_fs_ckpt_storage (./proton-data/checkpoint/), materialized_bytes=2
2026.01.07 04:42:04.211340 [ 5826 ] {.mv-8e9f4216-34cc-4547-b601-fcf461ffb70e} <Information> LocalFileSystemCheckpointStorage: Committed checkpoint 
epoch=3_01KEBC63H429WKM6YEXZSM1Z09 for query=.mv-8e9f4216-34cc-4547-b601-fcf461ffb70e

Epoch Old Version New Version
1 materialized rocks ckpt materialized rocks ckpt
2 materialized rocks ckpt materialized incremental rocks ckpt
3 materialized rocks ckpt materialized incremental rocks ckpt

The difference is the word "incremental" in the log message:

Old (broken):
RocksCheckpoint: Took 12 ms to materialized rocks ckpt to ...

New (fixed):
RocksCheckpoint: Took 10 ms to materialized incremental rocks ckpt to ...

The old version was ignoring the incremental=true setting and always doing full checkpoints. Now correctly enables incremental checkpointing for epoch 2+.

@yokofly
Copy link
Collaborator Author

yokofly commented Jan 7, 2026

@codex review

@chatgpt-codex-connector
Copy link

To use Codex here, create a Codex account and connect to github.

@yokofly
Copy link
Collaborator Author

yokofly commented Jan 7, 2026

@codex review

@yokofly yokofly marked this pull request as ready for review January 7, 2026 02:55
@yokofly yokofly merged commit 9841a56 into develop Jan 7, 2026
16 of 19 checks passed
@yokofly yokofly deleted the bugfix/issue-1079-honor-incremental-ckpt branch January 7, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

incremental checkpoint feature is implemented but never actually used

2 participants