API for external compressors

[M-Gonzalo]

I've been thinking about the mcm proposal, and how we sometimes need to test/work with different algorithms, especially in the early-ish stages of the project.

Would it be outlandish to suggest an integration to 3rd party programs as a plug-in final stage of compression? For example, let's say I want to use paq8px to compress my dwarfs blocks because I am a deranged person and who's gonna stop me. Or another more realistic example of course, like trying out a program as we did with mcm just to see how it compares to the existing solutions.

In practice, I would suggest something really simple for the plug-ins integration, like using pipes to a bash script that reads from stdin and writes to stdout. The script can even handle the special case of file-to-file compressors as it will, transparently, or you can build the code for handling temp files inside dwarfs too if you feel like it.

Thoughts @mhx?

[mhx]

This is actually on my (not written down anywhere) roadmap, but there are quite a few open points I need think about. I'll try to get back with a bit more detail when I'm not quite as busy. :)