Skip to content

Create throttler which is based on observing commit latency #468

@morgo

Description

@morgo

In move/buffered operations we currently use target-chunk-time to set the read-chunk-size. This is an MVCC read, so it doesn't provide the same benefit as in unbuffered where it yields locks. It does still yield read-views (which helps HLL), so it's not completely without purpose.

Writes are fanned out to 1000 rows/1MiB at a time, in write-threads. The default write threads is 2, which is low. Typically in production we will fan out to 32+ threads.

The bottlenecks shift from unbuffered operations:

  • We don't have contention on locks
  • Where we do have contention is the shared redo log.

So we should find a way to measure and yield based on that. Thus, my suggestion is a throttler based on observed commit latency.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions