Skip to content

Commit 7bc5dd9

Browse files
tbgtbg
committed
mmaprototype: add pre-means filtering for replica transfers
Previously, computeCandidatesForRange (now computeCandidatesForReplicaTransfer) computed the mean over all constraint-satisfying stores, then applied post-means exclusions before returning candidates. These exclusions were: - Stores on nodes already housing replicas of the range - The shedding store itself (handled via a separate check in the loop) - If the shedding store's node was CPU overloaded, all other stores on that node The last bullet was ad-hoc and asymmetric: we wouldn't apply the same logic to other overloaded nodes that happened to have replicas. It also wasn't exercised, and the downstream candidate selection logic wouldn't choose these stores anyway (they'd be rejected for being on an overloaded node). This commit makes two changes: 1. Simplified post-means exclusions: The parameter is renamed from storesToExclude to postMeansExclusions to clarify its role. All exclusions are now handled uniformly in this set: - The shedding store (we're moving away from it) - Stores on nodes with other existing replicas The ad-hoc "exclude other stores on shedding node if node is overloaded" logic is removed. Within-node rebalance (to another store on the shedding node) is now permitted at this stage. 2. Pre-means filtering: This is new. Before computing the mean, we now filter out stores that aren't ready to receive replicas based on disposition. As a legacy case, we manually filter on high disk utilization, but there is a TODO to fold this into the disposition as well - this is tracked and will be done separately. This is done by retainReadyReplicaTargetStoresOnly. The shedding store bypasses the disposition check since it already has the replica - its load should be in the mean regardless of its disposition for NEW replicas. When filtering occurs, we recompute the mean over the filtered set. The common case (nothing filtered) still uses the cached mean for efficiency. Other cleanup: - Fixed logging (VEventf instead of mixing V() and Infof) - Added scratch fields to clusterState for the new filtering but also left a TODO to get out of scratch fields entirely. Part of #156776. Epic: CRDB-55052
1 parent 3ce534a commit 7bc5dd9

11 files changed

+572
-59
lines changed

‎pkg/kv/kvserver/allocator/mmaprototype/allocator_state.go‎

Lines changed: 104 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -815,10 +815,10 @@ func (cs *clusterState) ensureAnalyzedConstraints(rstate *rangeState) {
815815
// - Need diversity change for each candidate.
816816
//
817817
// The first 3 bullets are encapsulated in the helper function
818-
// computeCandidatesForRange. It works for both replica additions and
818+
// computeCandidatesForReplicaTransfer. It works for both replica additions and
819819
// rebalancing.
820820
//
821-
// For the last bullet (diversity), the caller of computeCandidatesForRange
821+
// For the last bullet (diversity), the caller of computeCandidatesForReplicaTransfer
822822
// needs to populate candidateInfo.diversityScore for each candidate in
823823
// candidateSet. It does so via diversityScoringMemo. Then the (loadSummary,
824824
// diversityScore) pair can be used to order candidates for attempts to add.
@@ -846,39 +846,123 @@ func (cs *clusterState) ensureAnalyzedConstraints(rstate *rangeState) {
846846

847847
// loadSheddingStore is only specified if this candidate computation is
848848
// happening because of overload.
849-
func (cs *clusterState) computeCandidatesForRange(
849+
//
850+
// postMeansExclusions are filtered post-means: their load is included in the
851+
// mean (they're viable locations in principle) but they're not candidates for
852+
// this specific transfer (the classic case: already have a replica).
853+
func (cs *clusterState) computeCandidatesForReplicaTransfer(
850854
ctx context.Context,
851-
expr constraintsDisj,
852-
storesToExclude storeSet,
855+
conj constraintsConj,
856+
existingReplicas storeSet,
857+
postMeansExclusions storeSet,
853858
loadSheddingStore roachpb.StoreID,
854859
) (_ candidateSet, sheddingSLS storeLoadSummary) {
855-
means := cs.meansMemo.getMeans(expr)
856-
if loadSheddingStore > 0 {
857-
sheddingSS := cs.stores[loadSheddingStore]
858-
sheddingSLS = cs.meansMemo.getStoreLoadSummary(ctx, means, loadSheddingStore, sheddingSS.loadSeqNum)
859-
if sheddingSLS.sls <= loadNoChange && sheddingSLS.nls <= loadNoChange {
860-
// In this set of stores, this store no longer looks overloaded.
861-
return candidateSet{}, sheddingSLS
862-
}
860+
// Start with computing the stores (and corresponding means) that satisfy
861+
// the constraint expression. If we don't see a need to filter out any of
862+
// these stores before computing the means, we can use it verbatim, otherwise
863+
// we will recompute the means again below.
864+
cs.scratchDisj[0] = conj
865+
means := cs.meansMemo.getMeans(cs.scratchDisj[:1])
866+
867+
// Pre-means filtering: copy to scratch, then filter in place.
868+
// Filter out stores that have a non-OK replica disposition.
869+
cs.scratchStoreSet = append(cs.scratchStoreSet[:0], means.stores...)
870+
filteredStores := retainReadyReplicaTargetStoresOnly(ctx, cs.scratchStoreSet, cs.stores, existingReplicas)
871+
872+
// Determine which means to use.
873+
//
874+
// TODO(tbg): unit testing.
875+
var effectiveMeans *meansLoad
876+
if len(filteredStores) == len(means.stores) {
877+
// Common case: nothing was filtered, use cached means.
878+
effectiveMeans = &means.meansLoad
879+
} else if len(filteredStores) == 0 {
880+
// No viable candidates at all.
881+
return candidateSet{}, sheddingSLS
882+
} else {
883+
// Some stores were filtered; recompute means over filtered set.
884+
cs.scratchMeans = computeMeansForStoreSet(
885+
cs, filteredStores, cs.meansMemo.scratchNodes, cs.meansMemo.scratchStores)
886+
effectiveMeans = &cs.scratchMeans
887+
log.KvDistribution.VEventf(ctx, 2,
888+
"pre-means filtered %d stores → remaining %v, means: store=%v node=%v",
889+
len(means.stores)-len(filteredStores), filteredStores,
890+
effectiveMeans.storeLoad, effectiveMeans.nodeLoad)
863891
}
864-
// We only filter out stores that are not fdOK. The rest of the filtering
865-
// happens later.
892+
893+
sheddingSLS = cs.computeLoadSummary(ctx, loadSheddingStore, &effectiveMeans.storeLoad, &effectiveMeans.nodeLoad)
894+
if sheddingSLS.sls <= loadNoChange && sheddingSLS.nls <= loadNoChange {
895+
// In this set of stores, this store no longer looks overloaded.
896+
return candidateSet{}, sheddingSLS
897+
}
898+
866899
var cset candidateSet
867-
for _, storeID := range means.stores {
868-
if storesToExclude.contains(storeID) {
900+
for _, storeID := range filteredStores {
901+
if postMeansExclusions.contains(storeID) {
902+
// This store's load is included in the mean, but it's not a viable
903+
// target for this specific transfer (e.g. it already has a replica).
869904
continue
870905
}
871-
ss := cs.stores[storeID]
872-
csls := cs.meansMemo.getStoreLoadSummary(ctx, means, storeID, ss.loadSeqNum)
906+
csls := cs.computeLoadSummary(ctx, storeID, &effectiveMeans.storeLoad, &effectiveMeans.nodeLoad)
873907
cset.candidates = append(cset.candidates, candidateInfo{
874908
StoreID: storeID,
875909
storeLoadSummary: csls,
876910
})
877911
}
878-
cset.means = &means.meansLoad
912+
cset.means = effectiveMeans
879913
return cset, sheddingSLS
880914
}
881915

916+
// retainReadyReplicaTargetStoresOnly filters the input set to only those stores
917+
// that are ready to accept a replica. A store is not ready if it has a non-OK
918+
// replica disposition. In practice, the input set is already filtered by
919+
// constraints.
920+
//
921+
// Stores already housing a replica (on top of being in the input storeSet)
922+
// bypass this disposition check since they already have the replica - its load
923+
// should be in the mean regardless of its disposition, as we'll pick candidates
924+
// based on improving clustering around the mean.
925+
//
926+
// The input storeSet is mutated (and returned as the result).
927+
func retainReadyReplicaTargetStoresOnly(
928+
ctx context.Context,
929+
in storeSet,
930+
stores map[roachpb.StoreID]*storeState,
931+
existingReplicas storeSet,
932+
) storeSet {
933+
out := in[:0]
934+
for _, storeID := range in {
935+
if existingReplicas.contains(storeID) {
936+
// Stores on existing replicas already have the load and we want to
937+
// include them in the mean, even if they are not accepting new replicas
938+
// or even try to shed.
939+
//
940+
// TODO(tbg): health might play into this, though. For example, when
941+
// a store is dead, whatever load we have from it is stale and we
942+
// are better off not including it. For now, we ignore this problem
943+
// because the mma only handles rebalancing, whereas a replica on a
944+
// dead store would be removed by the single-metric allocator after
945+
// the TimeUntilStoreDead and so would disappear from our view.
946+
out = append(out, storeID)
947+
continue
948+
}
949+
ss := stores[storeID]
950+
switch {
951+
case ss.status.Disposition.Replica != ReplicaDispositionOK:
952+
log.KvDistribution.VEventf(ctx, 2, "skipping s%d for replica transfer: replica disposition %v (health %v)", storeID, ss.status.Disposition.Replica, ss.status.Health)
953+
case highDiskSpaceUtilization(ss.reportedLoad[ByteSize], ss.capacity[ByteSize]):
954+
// TODO(tbg): remove this from mma and just let the caller set this
955+
// disposition based on the following cluster settings:
956+
// - kv.allocator.max_disk_utilization_threshold
957+
// - kv.allocator.rebalance_to_max_disk_utilization_threshold
958+
log.KvDistribution.VEventf(ctx, 2, "skipping s%d for replica transfer: high disk utilization (health %v)", storeID, ss.status.Health)
959+
default:
960+
out = append(out, storeID)
961+
}
962+
}
963+
return out
964+
}
965+
882966
// Diversity scoring is very amenable to caching, since the set of unique
883967
// locality tiers for range replicas is likely to be small. And the cache does
884968
// not need to be cleared after every allocator pass. This caching is done via

‎pkg/kv/kvserver/allocator/mmaprototype/cluster_state.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1259,6 +1259,9 @@ type clusterState struct {
12591259
ranges map[roachpb.RangeID]*rangeState
12601260

12611261
scratchRangeMap map[roachpb.RangeID]struct{}
1262+
scratchStoreSet storeSet // scratch space for pre-means filtering
1263+
scratchMeans meansLoad // scratch space for recomputed means
1264+
scratchDisj [1]constraintsConj // scratch space for getMeans call
12621265

12631266
// Added to when a change is proposed. Will also add to corresponding
12641267
// rangeState.pendingChanges and to the affected storeStates.

‎pkg/kv/kvserver/allocator/mmaprototype/cluster_state_rebalance_stores.go‎

Lines changed: 35 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -61,12 +61,13 @@ type rebalanceEnv struct {
6161
// rebalanceStores invocation.
6262
now time.Time
6363
// Scratch variables reused across iterations.
64+
// TODO(tbg): these are a potential source of errors (imagine two nested
65+
// calls using the same scratch variable). Just make a global variable
66+
// that wraps a bunch of sync.Pools for the types we need.
6467
scratch struct {
65-
disj [1]constraintsConj
66-
storesToExclude storeSet
67-
storesToExcludeForRange storeSet
68-
nodes map[roachpb.NodeID]*NodeLoad
69-
stores map[roachpb.StoreID]struct{}
68+
postMeansExclusions storeSet
69+
nodes map[roachpb.NodeID]*NodeLoad
70+
stores map[roachpb.StoreID]struct{}
7071
}
7172
}
7273

@@ -367,22 +368,6 @@ func (re *rebalanceEnv) rebalanceReplicas(
367368
log.KvDistribution.VEventf(ctx, 2, "skipping remote store s%d: in lease shedding grace period", store.StoreID)
368369
return
369370
}
370-
// If the node is cpu overloaded, or the store/node is not fdOK, exclude
371-
// the other stores on this node from receiving replicas shed by this
372-
// store.
373-
excludeStoresOnNode := store.nls > overloadSlow
374-
re.scratch.storesToExclude = re.scratch.storesToExclude[:0]
375-
if excludeStoresOnNode {
376-
nodeID := ss.NodeID
377-
for _, storeID := range re.nodes[nodeID].stores {
378-
re.scratch.storesToExclude.insert(storeID)
379-
}
380-
log.KvDistribution.VEventf(ctx, 2, "excluding all stores on n%d due to overload/fd status", nodeID)
381-
} else {
382-
// This store is excluded of course.
383-
re.scratch.storesToExclude.insert(store.StoreID)
384-
}
385-
386371
// Iterate over top-K ranges first and try to move them.
387372
topKRanges := ss.adjusted.topKRanges[localStoreID]
388373
n := topKRanges.len()
@@ -428,6 +413,8 @@ func (re *rebalanceEnv) rebalanceReplicas(
428413
"rstate_replicas=%v rstate_constraints=%v",
429414
store.StoreID, rangeID, rstate.pendingChanges, rstate.replicas, rstate.constraints))
430415
}
416+
// Get the constraint conjunction which will allow us to look up stores
417+
// that could replace the shedding store.
431418
var conj constraintsConj
432419
var err error
433420
if isVoter {
@@ -441,31 +428,43 @@ func (re *rebalanceEnv) rebalanceReplicas(
441428
log.KvDistribution.VEventf(ctx, 2, "skipping r%d: constraint violation needs fixing first: %v", rangeID, err)
442429
continue
443430
}
444-
re.scratch.disj[0] = conj
445-
re.scratch.storesToExcludeForRange = append(re.scratch.storesToExcludeForRange[:0], re.scratch.storesToExclude...)
446-
// Also exclude all stores on nodes that have existing replicas.
431+
// Build post-means exclusions: stores whose load is included in the mean
432+
// (they're viable locations in principle) but aren't valid targets for
433+
// this specific transfer.
434+
//
435+
// NB: to prevent placing replicas on multiple CRDB nodes sharing a
436+
// host, we'd need to make changes here.
437+
// See: https://github.com/cockroachdb/cockroach/issues/153863
438+
re.scratch.postMeansExclusions = re.scratch.postMeansExclusions[:0]
439+
existingReplicas := storeSet{} // TODO(tbg): avoid allocation
447440
for _, replica := range rstate.replicas {
448441
storeID := replica.StoreID
442+
existingReplicas.insert(storeID)
449443
if storeID == store.StoreID {
450-
// We don't exclude other stores on this node, since we are allowed to
451-
// transfer the range to them. If the node is overloaded or not fdOK,
452-
// we have already excluded those stores above.
444+
// Exclude the shedding store (we're moving away from it), but not
445+
// other stores on its node (within-node rebalance is allowed).
446+
re.scratch.postMeansExclusions.insert(storeID)
453447
continue
454448
}
449+
// Exclude all stores on nodes with other existing replicas.
455450
nodeID := re.stores[storeID].NodeID
456451
for _, storeID := range re.nodes[nodeID].stores {
457-
re.scratch.storesToExcludeForRange.insert(storeID)
452+
re.scratch.postMeansExclusions.insert(storeID)
458453
}
459454
}
455+
456+
// Compute the candidates. These are already filtered down to only those stores
457+
// that we'll actually be happy to transfer the range to.
458+
// Note that existingReplicas is a subset of postMeansExclusions, so they'll
459+
// be included in the mean, but are never considered as candidates.
460+
//
460461
// TODO(sumeer): eliminate cands allocations by passing a scratch slice.
461-
cands, ssSLS := re.computeCandidatesForRange(ctx, re.scratch.disj[:], re.scratch.storesToExcludeForRange, store.StoreID)
462+
cands, ssSLS := re.computeCandidatesForReplicaTransfer(ctx, conj, existingReplicas, re.scratch.postMeansExclusions, store.StoreID)
462463
log.KvDistribution.VEventf(ctx, 2, "considering replica-transfer r%v from s%v: store load %v",
463464
rangeID, store.StoreID, ss.adjusted.load)
464-
if log.V(2) {
465-
log.KvDistribution.Infof(ctx, "candidates are:")
466-
for _, c := range cands.candidates {
467-
log.KvDistribution.Infof(ctx, " s%d: %s", c.StoreID, c.storeLoadSummary)
468-
}
465+
log.KvDistribution.VEventf(ctx, 3, "candidates are:")
466+
for _, c := range cands.candidates {
467+
log.KvDistribution.VEventf(ctx, 3, " s%d: %s", c.StoreID, c.storeLoadSummary)
469468
}
470469

471470
if len(cands.candidates) == 0 {
@@ -667,8 +666,9 @@ func (re *rebalanceEnv) rebalanceLeasesFromLocalStoreID(
667666
// leaseholder is always going to include itself in the mean, even if it
668667
// is ill-disposed towards leases.
669668
candsPL = retainReadyLeaseTargetStoresOnly(ctx, candsPL, re.stores, rangeID, store.StoreID)
670-
671669
clear(re.scratch.nodes)
670+
// NB: candsPL is not empty - it includes at least the current leaseholder
671+
// and one additional candidate.
672672
means := computeMeansForStoreSet(re, candsPL, re.scratch.nodes, re.scratch.stores)
673673
sls := re.computeLoadSummary(ctx, store.StoreID, &means.storeLoad, &means.nodeLoad)
674674
if sls.dimSummary[CPURate] < overloadSlow {

‎pkg/kv/kvserver/allocator/mmaprototype/cluster_state_test.go‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -419,6 +419,11 @@ func TestClusterState(t *testing.T) {
419419
ctx, finishAndGet := tracing.ContextWithRecordingSpan(
420420
context.Background(), tr, d.Cmd,
421421
)
422+
if d.HasArg("breakpoint") {
423+
// You can set a debugger breakpoint here and use `breakpoint=true`
424+
// in a datadriven command to hit it.
425+
t.Log("hit breakpoint")
426+
}
422427
switch d.Cmd {
423428
case "include":
424429
loc := dd.ScanArg[string](t, d, "path")
@@ -630,6 +635,19 @@ func TestClusterState(t *testing.T) {
630635
rec.SafeFormatMinimal(&sb)
631636
return fmt.Sprintf("%s%v\n", sb.String(), out)
632637

638+
case "retain-ready-replica-target-stores-only":
639+
in := dd.ScanArg[[]roachpb.StoreID](t, d, "in")
640+
replicas, _ := dd.ScanArgOpt[[]roachpb.StoreID](t, d, "replicas")
641+
var replicasSet storeSet
642+
for _, replica := range replicas {
643+
replicasSet.insert(replica)
644+
}
645+
out := retainReadyReplicaTargetStoresOnly(ctx, storeSet(in), cs.stores, replicasSet)
646+
rec := finishAndGet()
647+
var sb redact.StringBuilder
648+
rec.SafeFormatMinimal(&sb)
649+
return fmt.Sprintf("%s%v\n", sb.String(), out)
650+
633651
default:
634652
panic(fmt.Sprintf("unknown command: %v", d.Cmd))
635653
}

‎pkg/kv/kvserver/allocator/mmaprototype/testdata/cluster_state/rebalance_stores_cpu_lease_refusing_target.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,8 @@ rebalance-stores store-id=1
6161
[mmaid=1] skipping s2 for lease transfer: lease disposition refusing (health ok)
6262
[mmaid=1] result(failed): skipping r1 since store not overloaded relative to candidates
6363
[mmaid=1] attempting to shed replicas next
64-
[mmaid=1] excluding all stores on n1 due to overload/fd status
6564
[mmaid=1] considering replica-transfer r1 from s1: store load [cpu:1000, write-bandwidth:0, byte-size:0]
65+
[mmaid=1] candidates are:
6666
[mmaid=1] result(failed): no candidates found for r1 after exclusions
6767
[mmaid=1] start processing shedding store s3: cpu node load overloadUrgent, store load overloadUrgent, worst dim CPURate
6868
[mmaid=1] no top-K[CPURate] ranges found for s3 with lease on local s1

‎pkg/kv/kvserver/allocator/mmaprototype/testdata/cluster_state/rebalance_stores_cpu_replica_count.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,9 @@ rebalance-stores store-id=1 max-range-move-count=1 fraction-pending-decrease-thr
1616
[mmaid=2] start processing shedding store s3: cpu node load overloadUrgent, store load overloadUrgent, worst dim CPURate
1717
[mmaid=2] top-K[CPURate] ranges for s3 with lease on local s1: r3:[cpu:100, write-bandwidth:0, byte-size:0] r2:[cpu:100, write-bandwidth:0, byte-size:0] r1:[cpu:100, write-bandwidth:0, byte-size:0]
1818
[mmaid=2] attempting to shed replicas next
19-
[mmaid=2] excluding all stores on n3 due to overload/fd status
2019
[mmaid=2] considering replica-transfer r3 from s3: store load [cpu:1000, write-bandwidth:0, byte-size:0]
20+
[mmaid=2] candidates are:
21+
[mmaid=2] s4: (store=loadNormal worst=WriteBandwidth cpu=loadLow writes=loadNormal bytes=loadNormal node=loadLow high_disk=false frac_pending=0.00,0.00(true))
2122
[mmaid=2] sortTargetCandidateSetAndPick: candidates: s4(SLS:loadNormal, overloadedDimLoadSummary:loadLow), overloadedDim:CPURate, picked s4
2223
[mmaid=2] can add load to n4s4: true targetSLS[(store=loadNormal worst=WriteBandwidth cpu=loadLow writes=loadNormal bytes=loadNormal node=loadLow high_disk=false frac_pending=0.00,0.00(true))] srcSLS[(store=overloadUrgent worst=CPURate cpu=overloadUrgent writes=loadNormal bytes=loadNormal node=overloadUrgent high_disk=false frac_pending=0.00,0.00(true))]
2324
[mmaid=2] result(success): rebalancing r3 from s3 to s4 [change: r3=[change_replicas=[{ADD_VOTER n4,s4} {REMOVE_VOTER n3,s3}] cids=1,2]] with resulting loads source: [cpu:900, write-bandwidth:0, byte-size:0] target: [cpu:210, write-bandwidth:0, byte-size:0]

‎pkg/kv/kvserver/allocator/mmaprototype/testdata/cluster_state/rebalance_stores_cpu_replica_frac_threshold.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,9 @@ rebalance-stores store-id=1 max-range-move-count=999 fraction-pending-decrease-t
1616
[mmaid=2] start processing shedding store s3: cpu node load overloadUrgent, store load overloadUrgent, worst dim CPURate
1717
[mmaid=2] top-K[CPURate] ranges for s3 with lease on local s1: r3:[cpu:100, write-bandwidth:0, byte-size:0] r2:[cpu:100, write-bandwidth:0, byte-size:0] r1:[cpu:100, write-bandwidth:0, byte-size:0]
1818
[mmaid=2] attempting to shed replicas next
19-
[mmaid=2] excluding all stores on n3 due to overload/fd status
2019
[mmaid=2] considering replica-transfer r3 from s3: store load [cpu:1000, write-bandwidth:0, byte-size:0]
20+
[mmaid=2] candidates are:
21+
[mmaid=2] s4: (store=loadNormal worst=WriteBandwidth cpu=loadLow writes=loadNormal bytes=loadNormal node=loadLow high_disk=false frac_pending=0.00,0.00(true))
2122
[mmaid=2] sortTargetCandidateSetAndPick: candidates: s4(SLS:loadNormal, overloadedDimLoadSummary:loadLow), overloadedDim:CPURate, picked s4
2223
[mmaid=2] can add load to n4s4: true targetSLS[(store=loadNormal worst=WriteBandwidth cpu=loadLow writes=loadNormal bytes=loadNormal node=loadLow high_disk=false frac_pending=0.00,0.00(true))] srcSLS[(store=overloadUrgent worst=CPURate cpu=overloadUrgent writes=loadNormal bytes=loadNormal node=overloadUrgent high_disk=false frac_pending=0.00,0.00(true))]
2324
[mmaid=2] result(success): rebalancing r3 from s3 to s4 [change: r3=[change_replicas=[{ADD_VOTER n4,s4} {REMOVE_VOTER n3,s3}] cids=1,2]] with resulting loads source: [cpu:900, write-bandwidth:0, byte-size:0] target: [cpu:210, write-bandwidth:0, byte-size:0]

0 commit comments

Comments
 (0)