-
Notifications
You must be signed in to change notification settings - Fork 526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TBS: set sampling.tail.storage_limit to 70% of available disk space by default #15450
Conversation
This pull request does not have a backport label. Could you fix it @carsonip? 🙏
|
|
@simitt I've set it to backport-skip as I'm under the impression that it is 9.0 specific. |
if tailSamplingConfig.StorageLimitAuto { | ||
const ( | ||
// fallbackDefault is the default TBS storage limit if disk space detection fails. | ||
fallbackDefault = 3 << 30 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[to reviewer] It is still going to be an arbitrary number, but do we want to increase this to something like 5GB or 10GB?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the discussed solution, WDYT about changing the default to the % of total disk size, e.g. 70%.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there any chance that we would want to make "70%" configurable in the future too?
I doubt we need the configuration, as user can already configure the absolute number of storage e.g. "3GB", but I could be proven wrong in the long run. |
This pull request is now in conflicts. Could you fix it @carsonip? 🙏
|
StorageLimitParsed: 3000000000, | ||
StorageLimit: "", | ||
StorageLimitParsed: 0, | ||
StorageLimitAuto: true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not a huge fan of having two config options for the same concern.
Would it make sense to allow configuring either a fixed limit or a percentage in the StorageLimit
field? We could keep the 3GB
in 8.x
and then switch the default to 70%
in 9.0
.
This way the percentage would be configurable without introducing a new config and I assume some customers would prefer the percentage over the fixed size.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the idea. We can make StorageLimit accept a % suffix value. And the default for the StorageLimit is 70%. We will still need 2 fields: StorageLimitParsed
for an absolute value, StorageLimitPercentage
for the percentage. I'll proceed to implement this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The complexity with obtaining an absolute value from a passed in percentage is that ideally we'll add the current DB size to the available disk space, to avoid drifting as the db size increases. We either ignore the drift, or we use % of total size instead of available size. But we still have to determine beats data path here to calculate the storage limit, which may be a circular dependency.
Questions:
- When should we calculate the absolute value of storage limit - at config parse time, or at processor start time? Processor start time has the benefit of that beats data path is definitely correct and we'll have access to the db size.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Played around with it a little bit. What about doing the following:
- Move the logic for identifying the disk size limit to
internal/beater/config/sampling.go
and adding a config option for the storage dir:
// TailSamplingConfig holds configuration related to tail-sampling.
type TailSamplingConfig struct {
...
TTL time.Duration `config:"ttl" validate:"min=1s"`
StorageLimit string `config:"storage_limit"`
StorageLimitParsed uint64
StorageDir string
DiscardOnWriteFailure bool `config:"discard_on_write_failure"`
esConfigured bool
}
-
The diskspace calculation (including the dependency to libbeat paths) be moved to the
TailSamplingConfig
Unpack
function to calculate the available disk based on
(1) user specified GB limit
(2) total disk size * user configured percentage
(3) default to total disk size * 70%
Afaics this doesn't creaeete a circular dependency. -
Then only
cfg.StorageLimitParsed
andcfg.StorageDir
would need to be passed to the TBS implementation, everything else is isolated in the beater/config package.
sampling.tail.storage_limit
and storage limit handling
#14933
Superseded by #15467 |
Motivation/summary
Set
sampling.tail.storage_limit
to a 70% of available disk space by default when storage limit is unset. No behavior change if storage limit is set explicitly, either to 0 or non-0.Rebase after merging #15235
Checklist
For functional changes, consider:
How to test these changes
Start apm-server with
sampling.tail.storage_limit
unset, set to 0, and set to >0. In case of unset, check that storage limit is automatically set to a percentage of disk space by looking at the logs.Related issues
Fixes #14933