gc: improve the performance of Juicefs gc command #5683
Conversation
Improve the file deletion performance by processing multiple files in parallel ref: juicedata#5671
|
|
|
I have submitted a pull request for issue #5671, which aims to improve the performance of the gc command, especially when handling a large number of small files. I would appreciate it if someone could take a moment to review it. |
pkg/meta/tkv.go
Outdated
| startKey := m.fmtKey("D") | ||
| endKey := nextKey(startKey) | ||
| for { | ||
| keys, values, err := m.scan(startKey, endKey, batchSize, func(k, v []byte) bool { |
There was a problem hiding this comment.
use client.scan directly
There was a problem hiding this comment.
client.scan is a full scan method which retrieves all pending delete files, so it may not suitable for handling a large number of small files.
There was a problem hiding this comment.
The scan implementation in TiKV and FDB already handles this situation by fetching in batches, but etcd doesn’t. Let’s keep it like this then.
| return fmt.Errorf("invalid key %x", key) | ||
| for i := 0; i < threads; i++ { | ||
| wg.Add(1) | ||
| go func() { |
There was a problem hiding this comment.
If an exception occurs in the middle, the GC command should print that, same for redis and sql implement.
cmd/gc.go
Outdated
| @@ -70,6 +70,11 @@ $ juicefs gc redis://localhost --delete`, | |||
| Value: 10, | |||
| Usage: "number threads to delete leaked objects", | |||
| }, | |||
| &cli.IntFlag{ | |||
| Name: "cleanup-threads", | |||
There was a problem hiding this comment.
I think we can reuse the threads for cleanup
There was a problem hiding this comment.
OK, use the maxDeletes to delete objects and handle pending deleted file.
| deleteFileChan := make(chan redis.Z, threads) | ||
| var wg sync.WaitGroup | ||
|
|
||
| for i := 0; i < threads; i++ { |
There was a problem hiding this comment.
Can we move this part into base.go to reduce the duplicated code?
SonglinLife
force-pushed
the
enhance-gc
branch
from
43aac94 to
573111c
Compare
Improve the file deletion performance by processing multiple files in parallel
ref: #5671