Allow and prefer special vdevs as ZIL #17505
Conversation
There was a problem hiding this comment.
Conceptually, I think this makes sense. The special class is likely going to have better performance, and using that to store ZIL writes makes a lot of sense. The two parts that give me pause are:
- The special embedded log; this is a permanent sacrifice of some of the high-performing storage dedicated to storing metadata. For users with zil-heavy workloads, who don't also have a SLOG, it's probably worth it; but for users who don't fit that bill, they're losing some of their special space unconditionally. On the other hand, it's not that much space, all things considered. One metaslab (currently at most 16GiB) is not the end of the world.
- The attempt to automatically detect which class the write will eventually end up in. This sort of auto-sensing is very tricky, and can result in weird an unpredictable behavior once deployed. That said, for this case it's probably not too problematic if it gets it wrong; the data will still end up in the right place eventually.
Before this change ZIL blocks were allocated only from normal or SLOG vdevs. In typical situation when special vdevs are SSDs and normal are HDDs it could cause weird inversions when data blocks are written to SSDs, but ZIL referencing them to HDDs. This change assumes that special vdevs typically have much better (or at least not worse) latency than normal, and so in absence of SLOGs should store ZIL blocks. It means similar to normal vdevs introduction of special embedded log allocation class and updating the allocation fallback order to: SLOG -> special embedded log -> special -> normal embedded log -> normal. The code tries to guess whether data block is going to be written to normal or special vdev (it can not be done precisely before compression) and prefer indirect writes for blocks written to a special vdev to avoid double-write. For blocks that are going to be written to normal vdev, special vdev by default plays as SLOG, reducing write latency by the cost of higher special vdev wear, but it is tunable via module parameter. This should allow HDD pools with decent SSD as special vdev to work under synchronous workloads without requiring additional SLOG SSD, impractical in many scenarios. Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc.
True. I decided that it does not worth complexity of changing it in run time when SLOG added/removed, and may be not good for fragmentation to not reserve the log space. With my recent change we may allow fallback from
True. The only difference is whether we double-write when we should not or not when we should. The difference should be only in performance. |
| metaslab_class_t *spa_log_class; /* intent log data class */ | ||
| metaslab_class_t *spa_embedded_log_class; /* log on normal vdevs */ | ||
| metaslab_class_t *spa_special_class; /* special allocation class */ | ||
| metaslab_class_t *spa_special_embedded_log_class; /* log on special */ | ||
| metaslab_class_t *spa_dedup_class; /* dedup allocation class */ |
There was a problem hiding this comment.
Future cleanup opportunity: there's possibly enough of these now to make them an array indexed on an enum, and then we don't have to repeat ourselves over and over when doing something to all of them (init/fini, stats, etc).
Before this change ZIL blocks were allocated only from normal or SLOG vdevs. In typical situation when special vdevs are SSDs and normal are HDDs it could cause weird inversions when data blocks are written to SSDs, but ZIL referencing them to HDDs.
This change assumes that special vdevs typically have much better (or at least not worse) latency than normal, and so in absence of SLOGs should store ZIL blocks. It means similar to normal vdevs introduction of special embedded log allocation class and updating the allocation fallback order to: SLOG -> special embedded log -> special -> normal embedded log -> normal.
The code tries to guess whether data block is going to be written to normal or special vdev (it can not be done precisely before compression) and prefer indirect writes for blocks written to a special vdev to avoid double-write. For blocks that are going to be written to normal vdev, special vdev by default plays as SLOG, reducing write latency by the cost of higher special vdev wear, but it is tunable via module parameter.
This should allow HDD pools with decent SSD as special vdev to work under synchronous workloads without requiring additional SLOG SSD, impractical in many scenarios.
Types of changes
Checklist:
Signed-off-by.