This lambda function exists as an optional add-on for Oxbow when there are
large number of events being generated by S3 Event Notifications which share a
common prefix. For example, if an external system produces a substantial number
of PutObject calls for .parquet files with the prefix
databases/external/dw_sync/ this can result in the oxbow-lambda being
inveoked with high amounts of concurrency for the same Delta table, leading to
lock contention in AWS S3.
THe events grouping function aims to address this issue by acting as an
intermediary on the events pipeline and translating unordered SQS messages from
its trigger, into FIFO messages with a specific group identifier for an SQS
FIFO destination. By grouping on the prefix (e.g.
databases/external/dw_sync/) the oxbow-lambda can then be triggered from
the SQS FIFO queue where concurrent function invocations will have a far lower likelihood for lock contention.
Name |
Default Value |
Notes |
|
|
Set the log level, e.g. |
|
null |
Should only be used if S3 Event Notifications are first going to SNS and then routing to SQS for Oxbow |
|
null |
Optional integer value to subdivide the group further than just the hash of the previx and bucket |