Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python functions shut down slowly #109

Open
fernandezcuesta opened this issue Nov 15, 2024 · 4 comments · May be fixed by #130
Open

Python functions shut down slowly #109

fernandezcuesta opened this issue Nov 15, 2024 · 4 comments · May be fixed by #130
Labels
bug Something isn't working

Comments

@fernandezcuesta
Copy link
Contributor

fernandezcuesta commented Nov 15, 2024 •
edited
Loading

What happened?

I'm not fully sure where to report this, but the point is that after introducing a composition function developed with python, we noticed that:

  • The latency of the function deployment to do a rollout restart is way larger than others (e.g. based on go sdk)
  • The amount of time it takes for crossplane render got penalized as well.

This is negatively impacting both CI and DevEx since rendering a composition is quite a common use case.

How can we reproduce it?

I just tested this with a dummy function which does nothing:

    async def RunFunction(
        self, req: fnv1.RunFunctionRequest, _: grpc.aio.ServicerContext
    ) -> fnv1.RunFunctionResponse:
        """Run the function."""
        log = self.log.bind(tag=req.meta.tag)
        log.info("Running function and doing nothing")

        return response.to(req)

Running crossplane render on a composition that has this function as a dependency (not necessarily used in the pipeline, just declaring this as a dependency in crossplane.yaml is enough) adds an extra ~10s. A couple of experiments with simple compositions raise the amount of time for the render command from 1-2 seconds to 10-12 seconds.

I was suspecting that this could have something to do, and played by setting it to None but apparently didn't help :(

What environment did it happen in?

function-sdk-python version: 0.5.0
crank version v1.18.0
@fernandezcuesta fernandezcuesta added the bug Something isn't working label Nov 15, 2024
@fernandezcuesta
Copy link
Contributor Author

fernandezcuesta commented Nov 15, 2024 •
edited
Loading

Update: I verified that this doesn't happen with Crossplane <= v1.17.1, so both v1.17.2 and 1.18.0 are affected

@negz
Copy link
Member

negz commented Nov 21, 2024

I can't imagine why the version of Crossplane would affect function startup/shutdown time. Given it's Python I wouldn't ever expect it to be as fast as a Go function, but 10s startup does seem like a lot.

Are your tests factoring in image pull time? xpkg.upbound.io/negz/function-auto-python:v0.1.0 and xpkg.upbound.io/crossplane-contrib/function-auto-ready:v0.3.0 do the same thing, but the Python function is roughly twice as large. They both use Distroless image bases, so I'm guessing the extra size is the Python interpreter and stdlib etc.

@fernandezcuesta
Copy link
Contributor Author

fernandezcuesta commented Nov 25, 2024 •
edited
Loading

I'm afraid not, we took times when the image is already pulled. For instance the first time (fresh, so image being pulled) takes up to 26 seconds (for a regular crossplane render). I was also puzzled about the crossplane version being relevant but that surfaced from our CI pipelines right after upgrade of 1.17.1 to 1.18.0.

Btw the issue is definitely on shutdown. With debug logs I can see the function does its job quickly, then takes around 9s to quit, so essentially preventing the output of the render command to be shown.

fernandezcuesta added a commit to fernandezcuesta/function-sdk-python that referenced this issue Feb 17, 2025
@fernandezcuesta
fix: crossplane#109
4567ca0
fernandezcuesta added a commit to fernandezcuesta/function-sdk-python that referenced this issue Feb 17, 2025
@fernandezcuesta
fix: crossplane#109
869f7d4 
Signed-off-by: Jesús Fernández <[email protected]>
@fernandezcuesta fernandezcuesta linked a pull request Feb 17, 2025 that will close this issue
Open
2 tasks
@fernandezcuesta
Copy link
Contributor Author

I realized by chance that the problem comes on shutdown, regardless of the Crossplane version. What seems to be changed from >1.17.1 is the fact that the render command cleans up function containers on exit.
I created a PR in #130 to address this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: handle sigterm fernandezcuesta/function-sdk-python
2 participants
@negz @fernandezcuesta