Code

[Managed Iceberg] unbounded source#33504

Code

Open

ahmedabu98 wants to merge 36 commits intoapache:masterapache/beam:masterfrom ahmedabu98:iceberg_streaming_sourceahmedabu98/beam:iceberg_streaming_sourceCopy head branch name to clipboard

+3,411-369

Commits

Commits on Jan 6, 2025

• initial

  ahmedabu98

  committed
  bb87511

  Copy full SHA for bb87511

Commits on Jan 7, 2025

• let CombinedScanTask do splitting (based on Parquet row groups)

  ahmedabu98

  committed
  853de4d

  Copy full SHA for 853de4d

• perf improv

  ahmedabu98

  committed
  69fd988

  Copy full SHA for 69fd988

Commits on Jan 8, 2025

• create one read task descriptor per snapshot range

  ahmedabu98

  committed
  da2f33f

  Copy full SHA for da2f33f

Commits on Jan 22, 2025

• Merge branch 'master' of https://github.com/ahmedabu98/beam into iceberg_streaming_source

  ahmedabu98

  committed
  73c8992

  Copy full SHA for 73c8992

Commits on Jan 25, 2025

• some improvements

  ahmedabu98

  committed
  81ca709

  Copy full SHA for 81ca709

Commits on Jan 30, 2025

• use GiB for streaming, Redistribute for batch; update docs

  ahmedabu98

  committed
  e319d76

  Copy full SHA for e319d76

• Merge branch 'master' of https://github.com/ahmedabu98/beam into iceberg_streaming_source

  ahmedabu98

  committed
  c25cd75

  Copy full SHA for c25cd75

Commits on Feb 3, 2025

• use static value

  ahmedabu98

  committed
  af1ec85

  Copy full SHA for af1ec85

• add some test

  ahmedabu98

  committed
  f5d3268

  Copy full SHA for f5d3268

• Merge branch 'master' of https://github.com/ahmedabu98/beam into iceberg_streaming_source

  ahmedabu98

  committed
  df40239

  Copy full SHA for df40239

Commits on Feb 4, 2025

• Merge branch 'master' of https://github.com/ahmedabu98/beam into iceberg_streaming_source

  ahmedabu98

  committed
  43ab88f

  Copy full SHA for 43ab88f

Commits on Feb 7, 2025

• Merge branch 'master' of https://github.com/ahmedabu98/beam into iceberg_streaming_source

  ahmedabu98

  committed
  20db0ee

  Copy full SHA for 20db0ee

Commits on Feb 10, 2025

• add a java doc; don't use static block to create coder

  ahmedabu98

  committed
  622625f

  Copy full SHA for 622625f

• spotless

  ahmedabu98

  committed
  4c25d3f

  Copy full SHA for 4c25d3f

Commits on Feb 13, 2025

• add options: from/to timestamp, starting strategy, and streaming toggle

  ahmedabu98

  committed
  8666166

  Copy full SHA for 8666166

• trigger integration tests

  ahmedabu98

  committed
  297c309

  Copy full SHA for 297c309

• small test fix

  ahmedabu98

  committed
  5e3a2cc

  Copy full SHA for 5e3a2cc

Commits on Feb 14, 2025

• Merge branch 'master' of https://github.com/ahmedabu98/beam into iceberg_streaming_source

  ahmedabu98

  committed
  8b131fd

  Copy full SHA for 8b131fd

Commits on Feb 25, 2025

• scan every snapshot individually; use snapshot commit timestamp to mark progress; convert GiB output iterable to list because of RunnerV2 bug

  ahmedabu98

  committed
  887eff1

  Copy full SHA for 887eff1

Commits on Mar 3, 2025

• new schematransform for cdc streaming; add watermark configs

  ahmedabu98

  committed
  6cfc2d8

  Copy full SHA for 6cfc2d8

• cleanup

  ahmedabu98

  committed
  fbad86e

  Copy full SHA for fbad86e

• add guava import

  ahmedabu98

  committed
  50f9497

  Copy full SHA for 50f9497

• remove iceberg_cdc_read from xlang auto-wrapper gen

  ahmedabu98

  committed
  4f1f40b

  Copy full SHA for 4f1f40b

• fix javadoc

  ahmedabu98

  committed
  633365c

  Copy full SHA for 633365c

• cleanup

  ahmedabu98

  committed
  37485f1

  Copy full SHA for 37485f1

Commits on Mar 4, 2025

• spotless

  ahmedabu98

  committed
  4ede0e8

  Copy full SHA for 4ede0e8

• use CDC schema for batch and streaming; re-introduce boolean 'streaming' option; doc updates

  ahmedabu98

  committed
  db9fd63

  Copy full SHA for db9fd63

• add to CHANGES.md and discussion docs

  ahmedabu98

  committed
  79ab16a

  Copy full SHA for 79ab16a

• spotless

  ahmedabu98

  committed
  06a4cee

  Copy full SHA for 06a4cee

• Merge branch 'master' of https://github.com/ahmedabu98/beam into iceberg_streaming_source

  ahmedabu98

  committed
  132034f

  Copy full SHA for 132034f

Commits on Mar 5, 2025

• address review comments about java docs

  ahmedabu98

  committed
  795c87c

  Copy full SHA for 795c87c

• remove raw guava dep

  ahmedabu98

  committed
  c6461c9

  Copy full SHA for c6461c9

• add another test for read utils

  ahmedabu98

  committed
  7dbf3e1

  Copy full SHA for 7dbf3e1

Commits on Mar 6, 2025

• Merge branch 'master' of https://github.com/ahmedabu98/beam into iceberg_streaming_source

  ahmedabu98

  committed
  5263a13

  Copy full SHA for 5263a13

Commits on Mar 7, 2025

• use cached schemas

  ahmedabu98

  committed
  40fe4ab

  Copy full SHA for 40fe4ab