[XrdPfc] Add ResourceMonitor and Purge Plugin support #87
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…y() statement. Add example of a purge plugin.
…ring and purge manager. - Some code for path/lfn parsing and namespace traversal has been moved into separate files and refactored to allow for reuse in initial scanning, purge traversal, and, eventually, in user plugins. - DirState now holds full information about stats/changes/deltas during the current monitoring period as well as summed up information about current usage of resources in / under this directory. Both Stats and Usages are split into "here" part (in this directory only) and "recursive_subdir" part (summed up information from all the subdirs). - ResourceMonintor holds queues where file open/close/purge events are reported and periodically processed. At the same time, Stats from currently opened files are extracted. This is used for periodic updates of the DirState tree. - High-level processing in ResourceMonitor still needs to be finalized and put together with purge-plugin API. DirState tree will be periodically exported (snapshotted) into vector form and provided for monitoring (as g-stream or as on-disk binary-data, convertable to JSON) and space management (purge plugin).
- OpenRecord now contains bool telling ResMon if the file existed before. - DirState::find_path() and PathTokenizer can now return the last existing directory DirState in the hierarchy --> this allows ResMon to detect and report how many new subdirectories got created. - Cleanup queue processing code. - Start working on heart_beat() processing logick.
Avoid looping over all opened files to collect the stats every N-seconds. - ResourceMonitor: introduce also a Queue for Stats updates. - File: periodically report Stats updates to ResourceMonitor, when total bytes read accumulates to 2% of file size and on every Sync() to disk call. This could be furhter tuned at runtime as it is known if a file is over- or under-reporting within given update/processing time-windows.
- Includes export vector snapshot form. - Support writing of vector snapshot in JSON form (to be used in an outside utility program). - Some white-spaces fixes.
…ile-system block size.
- Separate snapshots for stat reporting and for purge. - ResourceMonitor::heart_beat() now drives the purge process. It still calls the old-style purge function -- but this can now be easily modularized as needed. - Uses stat-blocks as unit of direcotry / file usage.
…urceMonitor::initial_scan.
…. Remove non-supported command options from option parser (cut-n-pasted from create_file, apparently).
…ate fileusage limits in purge cycle when diskusage high water mark is not reached.
These are mostly things I determined I'd need as I was working on the LotMan plugin. The biggest thing was the addition of various extra public headers that I found I needed to import XrdPfc/XrdPfc.hh
Small updates encountered when working on Purge Plugin
…gin with /, but do not have to end with it.
…file usage limits are sets: base, nominal, and max
… number of bytes to remove by the default purge according to purge plugin.
…has a periodic call flag set. Pruge plugin has the period call set by the default. It can be overridden by the virtual XrdPfc::PurgePin::CallPeriodically() interface.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Proto PR to make it easier to synchronize development and discuss the implementation.
To be eventually rebased and made as a clean PR against xrootd/devel, for release 5.7.
Top-level overview
Some code for path/lfn parsing and namespace traversal has been moved into
separate files and refactored to allow for reuse in initial scanning, purge
traversal, and, eventually, in user plugins.
DirState now holds full information about stats/changes/deltas during the
current monitoring period as well as summed up information about current
usage of resources in / under this directory. Both Stats and Usages are
split into "here" part (in this directory only) and "recursive_subdir" part
(summed up information from all the subdirs).
ResourceMonintor holds queues where file open/close/purge events are
reported and periodically processed. At the same time, Stats from currently
opened files are extracted. This is used for periodic updates of the
DirState tree.
High-level processing in ResourceMonitor still needs to be finalized and put
together with purge-plugin API. DirState tree will be periodically exported
(snapshotted) into vector form and provided for monitoring (as g-stream or
as on-disk binary-data, convertable to JSON) and space management (purge
plugin).