Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ZSTD compression #28

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Add ZSTD compression #28

wants to merge 7 commits into from

Conversation

dbeinder
Copy link

@dbeinder dbeinder commented Jan 28, 2025

Picking up on #16

The first two commits integrate zstd compression into NSIS

  • Tested on MSVC 14 (Visual Studio 2022)
  • In my testing, zstd at level 9-19 beats zlib & bzip2 in compression ratio and speed
  • For the absolute best compression, lzma is still the best option, but everywhere else on the speed/size spectrum zstd seems to offer the best tradeoff with extremely wide tunability: https://facebook.github.io/zstd/
  • Zstd stubs are 90kiB, this could be reduced to 75kiB with minimal compile options
    • Decompression speed is >200MB/s on my machine, but this could probably be improved
    • Zstd decompression is always single-threaded, unless exehead is extended to decompressing multiple files in parallel
    • Zstd requires between 2-10MB memory depending on zstd level (1-19)
  • Compression in makensis is multithreaded
    • Compression speed at level 17 is 20MB/s on my machine
    • Without further changes, multithreading only helps for larger files or when using solid mode
    • Zstd compression uses 20MB per thread at level 9, and about 150MB per thread at level 19
  • I have made some minimal changes to zstd and added a diff similar to 7zip
    • One change is to use fallback byteswap functions instead of calling into the CRT
    • All other changes are #ifndef EXEHEAD to strip unused features and reduce code size of the stubs
  • The remaining uglyness is necessary to link zstd without a CRT into the stub
    • Zstd requires memcpy, memset, memmove, malloc, calloc, free, _allmul, and __allshl
    • For now, I have put simple implementations into exehead/crt_replacement.c
  • Documentation updated, zstd License (BSD) added
    • Documented SetCompressionLevel script command to control compression level for zstd (and zlib)
  • MakeNsisW & Zip2Exe updated

The newest 5 commits make everything Windows 95 compatible. Getting zstd to compile on VC6 would be a lot of work, and those changes would make updating zstd impractical - on the other hand, getting a EXE compiled by modern MSVC to run on Win95 is relatively easy:

  • For the stubs, patch the PE header to allow running on Win9x and avoid calls to ConditionVariable kernel API
  • For the Math plugin, instead of linking to a modern W95-incompatible CRT, embed libm and xprintf
  • Moving to modern MSVC only increases the size of the existing stubs by 1-2kiB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant