Skip to content

Commit cb82748

Browse files
byrnedjbyrnedj
committed
Background data movement (#20)
Background data movement using periodic workers. Attempts to evict/promote items per given thresholds for each class. These reduce p99 latency since there is a higher chance that an allocation slot is free in the tier we are allocating in.
1 parent 233be02 commit cb82748

24 files changed

+1283
-26
lines changed

‎MultiTierDataMovement.md

+90
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Background Data Movement
2+
3+
In order to reduce the number of online evictions and support asynchronous
4+
promotion - we have added two periodic workers to handle eviction and promotion.
5+
6+
The diagram below shows a simplified version of how the background evictor
7+
thread (green) is integrated to the CacheLib architecture.
8+
9+
<p align="center">
10+
<img width="640" height="360" alt="BackgroundEvictor" src="cachelib-background-evictor.png">
11+
</p>
12+
13+
## Background Evictors
14+
15+
The background evictors scan each class to see if there are objects to move the next (lower)
16+
tier using a given strategy. Here we document the parameters for the different
17+
strategies and general parameters.
18+
19+
- `backgroundEvictorIntervalMilSec`: The interval that this thread runs for - by default
20+
the background evictor threads will wake up every 10 ms to scan the AllocationClasses. Also,
21+
the background evictor thread will be woken up everytime there is a failed allocation (from
22+
a request handling thread) and the current percentage of free memory for the
23+
AllocationClass is lower than `lowEvictionAcWatermark`. This may render the interval parameter
24+
not as important when there are many allocations occuring from request handling threads.
25+
26+
- `evictorThreads`: The number of background evictors to run - each thread is a assigned
27+
a set of AllocationClasses to scan and evict objects from. Currently, each thread gets
28+
an equal number of classes to scan - but as object size distribution may be unequal - future
29+
versions will attempt to balance the classes among threads. The range is 1 to number of AllocationClasses.
30+
The default is 1.
31+
32+
- `maxEvictionBatch`: The number of objects to remove in a given eviction call. The
33+
default is 40. Lower range is 10 and the upper range is 1000. Too low and we might not
34+
remove objects at a reasonable rate, too high and it might increase contention with user threads.
35+
36+
- `minEvictionBatch`: Minimum number of items to evict at any time (if there are any
37+
candidates)
38+
39+
- `maxEvictionPromotionHotness`: Maximum candidates to consider for eviction. This is similar to `maxEvictionBatch`
40+
but it specifies how many candidates will be taken into consideration, not the actual number of items to evict.
41+
This option can be used to configure duration of critical section on LRU lock.
42+
43+
44+
### FreeThresholdStrategy (default)
45+
46+
- `lowEvictionAcWatermark`: Triggers background eviction thread to run
47+
when this percentage of the AllocationClass is free.
48+
The default is `2.0`, to avoid wasting capacity we don't set this above `10.0`.
49+
50+
- `highEvictionAcWatermark`: Stop the evictions from an AllocationClass when this
51+
percentage of the AllocationClass is free. The default is `5.0`, to avoid wasting capacity we
52+
don't set this above `10`.
53+
54+
55+
## Background Promoters
56+
57+
The background promoters scan each class to see if there are objects to move to a lower
58+
tier using a given strategy. Here we document the parameters for the different
59+
strategies and general parameters.
60+
61+
- `backgroundPromoterIntervalMilSec`: The interval that this thread runs for - by default
62+
the background promoter threads will wake up every 10 ms to scan the AllocationClasses for
63+
objects to promote.
64+
65+
- `promoterThreads`: The number of background promoters to run - each thread is a assigned
66+
a set of AllocationClasses to scan and promote objects from. Currently, each thread gets
67+
an equal number of classes to scan - but as object size distribution may be unequal - future
68+
versions will attempt to balance the classes among threads. The range is `1` to number of AllocationClasses. The default is `1`.
69+
70+
- `maxProtmotionBatch`: The number of objects to promote in a given promotion call. The
71+
default is 40. Lower range is 10 and the upper range is 1000. Too low and we might not
72+
remove objects at a reasonable rate, too high and it might increase contention with user threads.
73+
74+
- `minPromotionBatch`: Minimum number of items to promote at any time (if there are any
75+
candidates)
76+
77+
- `numDuplicateElements`: This allows us to promote items that have existing handles (read-only) since
78+
we won't need to modify the data when a user is done with the data. Therefore, for a short time
79+
the data could reside in both tiers until it is evicted from its current tier. The default is to
80+
not allow this (0). Setting the value to 100 will enable duplicate elements in tiers.
81+
82+
### Background Promotion Strategy (only one currently)
83+
84+
- `promotionAcWatermark`: Promote items if there is at least this
85+
percent of free AllocationClasses. Promotion thread will attempt to move `maxPromotionBatch` number of objects
86+
to that tier. The objects are chosen from the head of the LRU. The default is `4.0`.
87+
This value should correlate with `lowEvictionAcWatermark`, `highEvictionAcWatermark`, `minAcAllocationWatermark`, `maxAcAllocationWatermark`.
88+
- `maxPromotionBatch`: The number of objects to promote in batch during BG promotion. Analogous to
89+
`maxEvictionBatch`. It's value should be lower to decrease contention on hot items.
90+

‎cachelib/allocator/BackgroundMover-inl.h

+112
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
/*
2+
* Copyright (c) Intel and its affiliates.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
namespace facebook {
18+
namespace cachelib {
19+
20+
template <typename CacheT>
21+
BackgroundMover<CacheT>::BackgroundMover(
22+
Cache& cache,
23+
std::shared_ptr<BackgroundMoverStrategy> strategy,
24+
MoverDir direction)
25+
: cache_(cache), strategy_(strategy), direction_(direction) {
26+
if (direction_ == MoverDir::Evict) {
27+
moverFunc = BackgroundMoverAPIWrapper<CacheT>::traverseAndEvictItems;
28+
29+
} else if (direction_ == MoverDir::Promote) {
30+
moverFunc = BackgroundMoverAPIWrapper<CacheT>::traverseAndPromoteItems;
31+
}
32+
}
33+
34+
template <typename CacheT>
35+
BackgroundMover<CacheT>::~BackgroundMover() {
36+
stop(std::chrono::seconds(0));
37+
}
38+
39+
template <typename CacheT>
40+
void BackgroundMover<CacheT>::work() {
41+
try {
42+
checkAndRun();
43+
} catch (const std::exception& ex) {
44+
XLOGF(ERR, "BackgroundMover interrupted due to exception: {}", ex.what());
45+
}
46+
}
47+
48+
template <typename CacheT>
49+
void BackgroundMover<CacheT>::setAssignedMemory(
50+
std::vector<MemoryDescriptorType>&& assignedMemory) {
51+
XLOG(INFO, "Class assigned to background worker:");
52+
for (auto [tid, pid, cid] : assignedMemory) {
53+
XLOGF(INFO, "Tid: {}, Pid: {}, Cid: {}", tid, pid, cid);
54+
}
55+
56+
mutex.lock_combine([this, &assignedMemory] {
57+
this->assignedMemory_ = std::move(assignedMemory);
58+
});
59+
}
60+
61+
// Look for classes that exceed the target memory capacity
62+
// and return those for eviction
63+
template <typename CacheT>
64+
void BackgroundMover<CacheT>::checkAndRun() {
65+
auto assignedMemory = mutex.lock_combine([this] { return assignedMemory_; });
66+
67+
unsigned int moves = 0;
68+
std::set<ClassId> classes{};
69+
auto batches = strategy_->calculateBatchSizes(cache_, assignedMemory);
70+
71+
for (size_t i = 0; i < batches.size(); i++) {
72+
const auto [tid, pid, cid] = assignedMemory[i];
73+
const auto batch = batches[i];
74+
75+
classes.insert(cid);
76+
const auto& mpStats = cache_.getPoolByTid(pid, tid).getStats();
77+
78+
if (!batch) {
79+
continue;
80+
}
81+
82+
// try moving BATCH items from the class in order to reach free target
83+
auto moved = moverFunc(cache_, tid, pid, cid, batch);
84+
moves += moved;
85+
moves_per_class_[tid][pid][cid] += moved;
86+
totalBytesMoved.add(moved * mpStats.acStats.at(cid).allocSize);
87+
}
88+
89+
numTraversals.inc();
90+
numMovedItems.add(moves);
91+
totalClasses.add(classes.size());
92+
}
93+
94+
template <typename CacheT>
95+
BackgroundMoverStats BackgroundMover<CacheT>::getStats() const noexcept {
96+
BackgroundMoverStats stats;
97+
stats.numMovedItems = numMovedItems.get();
98+
stats.runCount = numTraversals.get();
99+
stats.totalBytesMoved = totalBytesMoved.get();
100+
stats.totalClasses = totalClasses.get();
101+
102+
return stats;
103+
}
104+
105+
template <typename CacheT>
106+
std::map<TierId, std::map<PoolId, std::map<ClassId, uint64_t>>>
107+
BackgroundMover<CacheT>::getClassStats() const noexcept {
108+
return moves_per_class_;
109+
}
110+
111+
} // namespace cachelib
112+
} // namespace facebook

‎cachelib/allocator/BackgroundMover.h

+103
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
/*
2+
* Copyright (c) Intel and its affiliates.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
#pragma once
18+
19+
#include "cachelib/allocator/BackgroundMoverStrategy.h"
20+
#include "cachelib/allocator/CacheStats.h"
21+
#include "cachelib/common/AtomicCounter.h"
22+
#include "cachelib/common/PeriodicWorker.h"
23+
24+
namespace facebook {
25+
namespace cachelib {
26+
27+
// wrapper that exposes the private APIs of CacheType that are specifically
28+
// needed for the cache api
29+
template <typename C>
30+
struct BackgroundMoverAPIWrapper {
31+
static size_t traverseAndEvictItems(C& cache,
32+
unsigned int tid,
33+
unsigned int pid,
34+
unsigned int cid,
35+
size_t batch) {
36+
return cache.traverseAndEvictItems(tid, pid, cid, batch);
37+
}
38+
39+
static size_t traverseAndPromoteItems(C& cache,
40+
unsigned int tid,
41+
unsigned int pid,
42+
unsigned int cid,
43+
size_t batch) {
44+
return cache.traverseAndPromoteItems(tid, pid, cid, batch);
45+
}
46+
};
47+
48+
enum class MoverDir { Evict = 0, Promote };
49+
50+
// Periodic worker that evicts items from tiers in batches
51+
// The primary aim is to reduce insertion times for new items in the
52+
// cache
53+
template <typename CacheT>
54+
class BackgroundMover : public PeriodicWorker {
55+
public:
56+
using Cache = CacheT;
57+
// @param cache the cache interface
58+
// @param strategy the stragey class that defines how objects are
59+
// moved,
60+
// (promoted vs. evicted and how much)
61+
BackgroundMover(Cache& cache,
62+
std::shared_ptr<BackgroundMoverStrategy> strategy,
63+
MoverDir direction_);
64+
65+
~BackgroundMover() override;
66+
67+
BackgroundMoverStats getStats() const noexcept;
68+
std::map<TierId, std::map<PoolId, std::map<ClassId, uint64_t>>>
69+
getClassStats() const noexcept;
70+
71+
void setAssignedMemory(
72+
std::vector<MemoryDescriptorType>&& assignedMemory);
73+
74+
private:
75+
std::map<TierId, std::map<PoolId, std::map<ClassId, uint64_t>>>
76+
moves_per_class_;
77+
// cache allocator's interface for evicting
78+
using Item = typename Cache::Item;
79+
80+
Cache& cache_;
81+
std::shared_ptr<BackgroundMoverStrategy> strategy_;
82+
MoverDir direction_;
83+
84+
std::function<size_t(
85+
Cache&, unsigned int, unsigned int, unsigned int, size_t)>
86+
moverFunc;
87+
88+
// implements the actual logic of running the background evictor
89+
void work() override final;
90+
void checkAndRun();
91+
92+
AtomicCounter numMovedItems{0};
93+
AtomicCounter numTraversals{0};
94+
AtomicCounter totalClasses{0};
95+
AtomicCounter totalBytesMoved{0};
96+
97+
std::vector<MemoryDescriptorType> assignedMemory_;
98+
folly::DistributedMutex mutex;
99+
};
100+
} // namespace cachelib
101+
} // namespace facebook
102+
103+
#include "cachelib/allocator/BackgroundMover-inl.h"

‎cachelib/allocator/BackgroundMoverStrategy.h

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
/*
2+
* Copyright (c) Facebook, Inc. and its affiliates.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
#pragma once
18+
19+
#include "cachelib/allocator/Cache.h"
20+
21+
22+
namespace facebook {
23+
namespace cachelib {
24+
25+
struct MemoryDescriptorType {
26+
MemoryDescriptorType(TierId tid, PoolId pid, ClassId cid) :
27+
tid_(tid), pid_(pid), cid_(cid) {}
28+
TierId tid_;
29+
PoolId pid_;
30+
ClassId cid_;
31+
};
32+
33+
// Base class for background eviction strategy.
34+
class BackgroundMoverStrategy {
35+
public:
36+
virtual std::vector<size_t> calculateBatchSizes(
37+
const CacheBase& cache,
38+
std::vector<MemoryDescriptorType> acVec) = 0;
39+
};
40+
41+
} // namespace cachelib
42+
} // namespace facebook

‎cachelib/allocator/CMakeLists.txt

+1
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ add_library (cachelib_allocator
3535
CCacheManager.cpp
3636
ContainerTypes.cpp
3737
FreeMemStrategy.cpp
38+
FreeThresholdStrategy.cpp
3839
HitsPerSlabStrategy.cpp
3940
LruTailAgeStrategy.cpp
4041
MarginalHitsOptimizeStrategy.cpp

‎cachelib/allocator/Cache.h

+6
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,12 @@ class CacheBase {
9898
//
9999
// @param poolId The pool id to query
100100
virtual const MemoryPool& getPool(PoolId poolId) const = 0;
101+
102+
// Get the reference to a memory pool using a tier id, for stats purposes
103+
//
104+
// @param poolId The pool id to query
105+
// @param tierId The tier of the pool id
106+
virtual const MemoryPool& getPoolByTid(PoolId poolId, TierId tid) const = 0;
101107

102108
// Get Pool specific stats (regular pools). This includes stats from the
103109
// Memory Pool and also the cache.

0 commit comments

Comments
 (0)