Ability for PIE to download pre-compiled binaries on Linux/OSX/etc

[godruoyi]

Description

Thank you for creating such a nice tool that allows us to avoid using PECL any more.

I would like to ask if there are any plans to support the direct installation of extension files like .dylib, .dll, and .so, without the need for building them when we run pie install . It would be simpler since we can build the extension in another language and publish the artifacts to GitHub, so that we can install them directly without having to build them again. I would be interested in supporting this feature if it is feasible.

[asgrim]

There is upsides and downsides to this approach:

First up regarding Linux/Mac (.so) binaries: Packages providing binaries would become responsible for building binaries for every architecture/system, which is a huge undertaking requiring massive CI pipelines to produce all the necessary permutations of the binaries for different platforms; consider that you have arm64, amd64, x86, as common ones, but there are also many other less common architectures. Additionally, you may get dependent library compatibility issues; for example glibc vs musl, or even just dependent libraries that may have existed on the CI builder, but don't actually exist on the end user's machine. I really do think this maintenance burden on extension maintainers is too much; it might be okay for some, but I certainly would not want the responsibility of maintaining that! This is the reason PIE (and PECL before it) compiles binaries on the end user's system; to ensure everything is compatible for their system.

Secondly, regarding Windows (.dll) binaries: PIE compatible extensions on Windows must already provide binaries. This has historically always been precompiled on PECL. This used to be done by a "Windows PECL build server"; once an extension release was uploaded to the PECL site, a build job to produce the Windows binaries was run. This is a MUCH simpler feat than the Linux ecosystem, due to the enormously fewer architecture and dependency issues. For PIE, we take a similar approach, but the responsibility for publishing Windows binaries moves to the extension maintainers, instead of PECL website. Thankfully, at least on GitHub, this process is very simple thanks to @shivammathur 's excellent work on the php/php-windows-builder GitHub action.

I am not sure how this can be workable or practical for Linux/Mac systems, unfortunately.

[remicollet]

I can't imagine how this can be possible

There are so many distributions, versions, php versions, installation modes, providers, and architectures, I can only imagine how the matrix will be huge.

Only for Fedora/RHEL, each release commonly means >100 builds (for my repo)

P.S. to summarize my thinking

• People building PHP from sources should be able to build any extension from sources (using pecl, pie...)

• People using a binary installation of PHP should find the binary of wanted extensions from the same provider (ex: using the distribution package manager)

[Xuanwo]

Hi, maybe we can follow python's approach that provides pre-built binaries as best efforts and fallback to build locally if not found.

In this way, we can build extensions for most platforms like {linux,mac,windows} * {x86_64, aarch64}.

[godruoyi]

Thank you, @asgrim and @remicollet for your reply, but I still think this approach is feasible, here are my options:

1. PIE is a tool designed to install PHP extensions. While it primarily facilitates installation from source, we can also add functionality that allows users to install PHP extensions from binary driectly(like github releases). This way, we don't need to rely on extension maintainers providing binaries for all platforms or versions. Instead, if the user explicitly specifies that want to install from binary without build from course, PIE can attempt to download it first directly.

2. As @Xuanwo mentioned, extension maintainers typically only publish binaries for mainstream operating systems and versions. If a specific OS or version isn't supported by prebuilt binaries, we can fall back on building the extension from source as needed.

Implements detail may like:

• extension maintainers publish pre-built binaries to github like
  • opeadal-0.1.0-php8.1-maxosx_10_12. dylib
  • opeadal-0.1.0-php8.2-maxosx_10_12. dylib
  • opeadal-0.1.0-php8.2-manylinux_2_28.so
• pie install opendal/opeadal:0.1.0 --binary
• match pre-built binary base on local os(linux,mac,etc) and php version
• if found, install it directly, if not, back to build from source.

what do you guys think, pls Let me know your thoughts!

[remicollet]

opeadal-0.1.0-php8.2-manylinux_2_28.so

Can't work, too many dependencies on too many different library versions (libc which you probably refer in "2_28" part is not enough)
Generic linux build is a fantasy

[godruoyi]

@remicollet do you have more context why PyPI can works fine when user want install smme package from pre-built binaries directly without build, you can see there are many pre-buiild binaries https://pypi.org/project/opendal/#files.

[remicollet]

@remicollet do you have more context why PyPI can works fine when user want install smme package from pre-built binaries directly without build, you can see there are many pre-buiild binaries https://pypi.org/project/opendal/#files.

As I already said "too many dependencies on too many different library versions"
Can work with bundling everything, but this sounds terrible.

As I also already said "People using a binary installation of PHP should find the binary of wanted extensions from the same provider "

[Radiergummi]

I don't think this topic should be left just like this. The amount of repeated builds of the same source code in modern container environments is horrendous, and wastes incredible amounts of compute for the same result.

PHP extensions don't need to be built for Itanium processors with obscure libc anymore; following the Pareto principle and building for AMD64/ARM64, the currently actively supported PHP versions, MacOS/Windows/the top three Linux distributions leaves you with 2x2x5 combinations per extension.

That is the approach pypi follows, and wheels can be downloaded almost always in favor to building fresh.
This is more responsible, both economically and ecologically.

[godruoyi]

I'm disappointed that PIE doesn't support this. Since we've already decided to develop a new PHP extension installer, why not draw inspiration from existing implementations like PyPI instead of recreating PECL from scratch in PHP and that need build every times when install a new extension? I sincerely hope we can reconsider this approach for the feature.

[asgrim]

Folks; this isn't off the table. I have some ideas to try out, but we need to ensure the stability and folks switching from PECL is all working okay, since PECL is now deprecated.

[godruoyi]

Thank you, @asgrim! Great to hear that.

[mokraemer]

@remicollet How do you build pie packages for fedora? The build system does not allow to download things during build.
On the other hand I guess you don't want to have specs which only contain download links and make the compilation on each package installation. At least this would require many devel packages, gcc, .... which is now not needed for anyone installing pecl packages on fedora and other distros.

[remicollet]

@remicollet How do you build pie packages for fedora? The build system does not allow to download things during build.

There is no change from pecl package. PIE command is not used (as PECL command was not used)

[realFlowControl]

Just chiming in here for Datadog. We ship the source, but also pre-build binaries via GitHub releases. To "solve" the glibc version problem we build with CentOS7 (glibc 2.17) for glibc binaries ... I do appreciate that discussion going forward 😉
So far, as someone in this discussion already mentioned our build matrix is basically:

• Windows, Linux glibc and Linux musl
• aarch64 and x86_64
• PHP 7.0 to 8.5
• NTS and ZTS

So far this suites basically everyone.
Most likely for this feature we need a fallback to compile from source in case the binary is not where it is expected

[asgrim]

Thank you @realFlowControl that is very helpful 👍

[asgrim]

This is being worked on 🎉 you may track into #436

[tillkruss]

@asgrim Wonderful, stoked about this!

It be amazing if the pie could also run a script after installing pre-compiled binary. Not sure if it's outside the scope.

[realFlowControl]

👀 isn't the current npm problem about this?
I'd propose to split this into a separate work stream

[asgrim]

@asgrim Wonderful, stoked about this!

It be amazing if the pie could also run a script after installing pre-compiled binary. Not sure if it's outside the scope

I'd say it's out of scope for this particular change, but since we use Composer under the hood, it should be fairly straightforward to add post install scripts in the same way? Would be great if you could add a new Idea discussion please, and we can spec something out!