Skip to content

Build tooling to move tasks from celery to taskworkers #85394

@markstory

Description

@markstory

In order to move workloads away from celery onto taskbrokers, we'll need a way to change which system a task activation is produced to at runtime outside of deploys.

Ideally we have an rollout tooling that:

  1. Allows us to shift load at the task level from celery to taskworkers.
  2. In the case of failures, activation creation can be shifted back to celery.
  3. Does not require application code deploys to change where activations go.

This tooling should require minimal application code changes, and shouldn't involve changing all of the callsites for a task, but changing task definitions is an option.

The design doc has a rough idea, but we'll likely need to refine it.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions