feat(datalayer): configurable per-source scrape interval - #2214
Conversation
Polling sources shared one Collector tick (~50ms), so slow-changing sources (DCGM, /v1/models) were scraped far more often than useful. Schedule each dataLayer.sources entry by a tick multiple of --refresh-metrics-interval; omit keeps every-tick behavior. Signed-off-by: noalimoy <nlimoy@redhat.com>
397e674 to
8a0ec35
Compare
Signed-off-by: noalimoy <nlimoy@redhat.com>
8a0ec35 to
97fcf9e
Compare
|
|
||
| Polling sources share one Collector goroutine per endpoint. The base tick is | ||
| `--refresh-metrics-interval` (default 50ms). Each polling source plugin accepts an | ||
| `interval` parameter (e.g. `"1s"`) that must be a positive multiple of the base tick; |
There was a problem hiding this comment.
nit:
- does not have to be exact multiple. we can do the math of
round-to-nearest(duration/base)in the data layer runtime. In the doc just say that the interval will be rounded to the nearest multiple of base. - I'm fine with keeping it as metrics refresh, but it isn't really tied to metrics anymore. Just something to consider. Rename and deprecation of the flag is not worth it, IMHO.
Q:
how do you propose to treat cases where interval < base tick? Right now user needs to be aware and configure correctly. Two options (at least): (1) pick smallest interval as base period (not necessarily --refresh-metrics-interval) and define all others as multiple (2) use --refresh-metrics-interval as base frequency, everything else runs off of it - log a warning if an interval is below and operate the lower values at the base frequency.
I prefer the first option.
side note: lets define a minimal base interval (50ms?) and log a warning if user sets lower.
| plugin.Plugin | ||
| Dispatch(ctx context.Context, ep Endpoint) error | ||
| AppendExtractor(ext plugin.Plugin) error | ||
| Interval() time.Duration |
There was a problem hiding this comment.
Q/nit:
another option would have been to add a (derived) interface with an additional Interval() method so that only interested collectors need to implement it and not need to add Interval() time.Duration { return 0 } where it is not needed. The runtime can assert and query the interval of only needed implementation.
WDYT?
Not pushing for it - looking for feedback.
| if cfg.UseNodeAddress { | ||
| opts = append(opts, http.WithUseNodeAddress()) | ||
| } | ||
| if intervalOpt, err := http.ParseIntervalOption(cfg.Interval); err != nil { |
There was a problem hiding this comment.
is there a parse duration in the standard lib that would work?
| - `insecureSkipVerify` (bool, optional, default: `true`): Skip TLS certificate verification. | ||
| - `caCertPath` (string, optional): PEM CA bundle to verify the target's server cert. | ||
| - `clientCertPath` / `clientKeyPath` (string, optional): client certificate for mTLS. Set both together. | ||
| - `interval` (string, optional): Scrape period (e.g. `"5s"`). Must be a positive |
There was a problem hiding this comment.
nit: the same help string appears in multiple locations. Would it be possible to have it specified once (e.g., might need a data layer README)?
What type of PR is this?
/kind feature
What this PR does / why we need it:
All polling data-layer sources share one Collector goroutine per endpoint and one base tick (
--refresh-metrics-interval, default 50ms). That cadence fits latency-sensitive model-server metrics, but wastes HTTP traffic for sources that update more slowly (e.g. DCGM ~1s,/v1/models).Each polling source plugin now accepts an optional
intervalparameter (alongsidescheme,path,port, etc.). ThePollingDispatcherinterface exposesInterval() time.Duration; the Runtime converts it to a period in base ticks (must be a positive multiple of the base tick) and the Collector dispatches a source only when due. Omittingintervalkeeps every-tick behavior.Scheduling uses a tick-based next-due check in the existing Collector loop. A full timing wheel was not needed for the small number of polling sources per endpoint; we can revisit if that grows.
Key design choices:
intervallives in pluginparameters— it is source-specific configuration, same as port or schemeParseIntervalOptioncentralises duration parsing so factories stay one-linersNewHTTPDataSourcerejects negative intervals at construction timeExample:
Which issue(s) this PR fixes:
Fixes #1880
Test plan
interval: 1sas 20 period ticks when base is 50msmake buildandmake testpass with zero failuresRelease note (write
NONEif no user-facing change):