Fast, hardware-accelerated CRC calculation in PHP for all known CRC-32 and CRC-64 variants using SIMD
intrinsics, which can exceed 100GiB/s for CRC-32, and 50GiB/s for CRC-64, on modern systems.
It is much, much faster (up to >200X) than the native crc32, crc32b, and crc32c
implementations in PHP, plus adds many more variants
(particularly CRC-64/NVME).
The performance gains are especially pronounced on aarch64 (Arm) systems, since PHP doesn't
currently use hardware acceleration there.
CRC-64/NVME is in use in a variety of large-scale and mission-critical systems, software, and hardware, such as:
- The
AWS S3recommended checksum - The Linux kernel
- The NVMe specification
hash('crc32') in PHP is not the same as crc32() in PHP, and doesn't match the crc32 definition or output in
many other programming languages and implementations (which is typically
CRC-32/ISO-HDLC)
Instead, it's actually CRC-32/BZIP2 which
is then byte-reversed. This extension provides a CrcFast\CRC_32_PHP algorithm const that performs the same calculation,
but accelerated.
$checksum = CrcFast\checksum(
algorithm: CrcFast\CRC_32_PHP,
string: '123456789'
); // 181989fc- simdutf PHP extension for Unicode validation and transcoding at billions of characters per second using the simdutf project.
- simdjson_plus PHP extension for parsing gigabytes of JSON per second using the simdjson project.
Uses the crc_fast Rust package and its C-compatible shared library
under the hood, so you'll need to have built and installed it. (See Usage, below).
See the change log.
Use Composer to install this library using PIE (note the Requirements above):
composer install awesome/crc-fastIf you're using a non-standard installation location for the crc_fast
library, you may need to specify where the crc_fast header (in include) and shared library (in lib) can be found:
composer install awesome/crc-fast --with-crc-fast=/path/to/crc-fastLike most PHP extensions, you can also build yourself:
phpize
./configure
makeor with a custom crc_fast location where the crc_fast header and shared
library can be found:
phpize
./configure --with-crc-fast=/path/to/crc-fast
makeExamples are for CRC-64/NVME, but you can use any supported algorithm
variant.
// calculate the checksum of a string
$checksum = CrcFast\hash(
algorithm: CrcFast\CRC_64_NVME,
string: '123456789'
); // ae8b14860a799888
// calculate the checksum of a file, which will chunk through the file optimally,
// limiting RAM usage and maximizing throughput
$checksum = CrcFast\hash_file(
algorithm: CrcFast\CRC_64_NVME,
filename: 'path/to/123456789.txt',
); // ae8b14860a799888$crc64Digest = CrcFast\Digest::new(
algorithm: CrcFast\CRC_64_NVME,
);
// write some data to the digest
$crc64Digest->write('1234');
// write some more data to the digest
$crc64Digest->write('56789');
// calculate the entire digest
$checksum = $crc64Digest->finalize(); // ae8b14860a799888$algorithms = get_supported_algorithms();
var_dump($algorithms);Calculates CRC-32/ISO-HDLC as an integer.
$checksum = CrcFast\crc32(
data: '123456789'
); // 3421780262Calculates CRC-32/PHP as binary or hex.
$checksum = CrcFast\hash(
algorithm: CrcFast\CRC_32_PHP,
string: '123456789'
binary: false,
); // 181989fcCalculates CRC-32/ISO-HDLC as binary or hex.
$checksum = CrcFast\hash(
algorithm: CrcFast\CRC_32_ISO_HDLC,
string: '123456789'
binary: false,
); // cbf43926Calculates CRC-32/ISCSI as binary or hex.
$checksum = CrcFast\hash(
algorithm: CrcFast\CRC_32_ISCSI,
string: '123456789'
binary: false,
); // b798b438This extension comes with IDE stubs for use with your favorite development environment.
See the tests directory for test coverage, which also double as useful examples.
make testThis extension has been extensively tested on macOS and Linux, on both aarch64 and x86_64.
At Awesome we use it in production at very large scale on Linux on both
Flickr and SmugMug.
This extension is not currently supported on Windows. :(
The underlying crc_fast library (same authors) builds and works on
Windows, so this is likely just a build issue with creating a working config.w32 implementation. (I took a quick
stab, failed, and moved on since we don't use Windows in production.)
If you want to help, please open a working PR. I'd love to merge it.
PHP already uses SIMD intrinsics for CRC-32 calculations on x86_64 but not on aarch64. Even on x86_64, this
library provides considerable improvements, in addition to supporting many more variants.
Tested using the maximum settings for crc_fast for each platform, using 1MiB random payloads.
| Arch | Brand | CPU | System | PHP | crc_fast | Speedup |
|---|---|---|---|---|---|---|
| x86_64 | Intel | Sapphire Rapids | EC2 c7i.metal-48xl | ~27.0 GiB/s | ~108.1 GiB/s | ~4X |
| x86_64 | AMD | Genoa | EC2 c7a.metal-48xl | ~13.6 GiB/s | ~53.7 GiB/s | ~4X |
| aarch64 | AWS | Graviton4 | EC2 c8g.metal-48xl | ~0.4 GiB/s | ~52.3 GiB/s | ~141X |
| aarch64 | Apple | M3 Ultra | Mac Studio (32 core) | ~0.4 GiB/s | ~99.6 GiB/s | ~233X |
| Arch | Brand | CPU | System | PHP | crc_fast | Speedup |
|---|---|---|---|---|---|---|
| x86_64 | Intel | Sapphire Rapids | EC2 c7i.metal-48xl | ~26.6 GiB/s | ~27.4 GiB/s | n/a |
| x86_64 | AMD | Genoa | EC2 c7a.metal-48xl | ~13.6 GiB/s | ~25.4 GiB/s | ~2X |
| aarch64 | AWS | Graviton4 | EC2 c8g.metal-48xl | ~0.4 GiB/s | ~31.5 GiB/s | ~73X |
| aarch64 | Apple | M3 Ultra | Mac Studio (32 core) | ~0.4 GiB/s | ~57.8 GiB/s | ~134X |
Note that PHP has no native equivalent.
| Arch | Brand | CPU | System | crc_fast |
|---|---|---|---|---|
| x86_64 | Intel | Sapphire Rapids | EC2 c7i.metal-48xl | ~54.6 GiB/s |
| x86_64 | AMD | Genoa | EC2 c7a.metal-48xl | ~27.0 GiB/s |
| aarch64 | AWS | Graviton4 | EC2 c8g.metal-48xl | ~37.0 GiB/s |
| aarch64 | Apple | M3 Ultra | Mac Studio (32 core) | ~70.0 GiB/s |
cfc-fast is dual-licensed under
- Apache 2.0 license (LICENSE-Apache or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)