Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build tooling to move tasks from celery to taskworkers #85394

Open
markstory opened this issue Feb 18, 2025 · 0 comments
Open

Build tooling to move tasks from celery to taskworkers #85394

markstory opened this issue Feb 18, 2025 · 0 comments

Comments

@markstory
Copy link
Member

In order to move workloads away from celery onto taskbrokers, we'll need a way to change which system a task activation is produced to at runtime outside of deploys.

Ideally we have an rollout tooling that:

  1. Allows us to shift load at the task level from celery to taskworkers.
  2. In the case of failures, activation creation can be shifted back to celery.
  3. Does not require application code deploys to change where activations go.

This tooling should require minimal application code changes, and shouldn't involve changing all of the callsites for a task, but changing task definitions is an option.

The design doc has a rough idea, but we'll likely need to refine it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant