Skip to content

Sync buildkite builds and jobs dynamoDB tables to ClickHouse #7002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

huydhn
Copy link
Contributor

@huydhn huydhn commented Aug 13, 2025

A follow-up of #6998 and #7001 to sync these dynamoDB tables to ClickHouse, I will need to find a permanent home for these tables instead of using fortesting

Testing

Actually running and ingesting data at the moment (I update the lambda manually on AWS). Also backfilling the data from DynamoDB as I have run a debug version of #6998 since mid July.

python dynamo2ch.py --dynamodb-table vllm-buildkite-build-events --clickhouse-table fortesting.vllm_buildkite_builds --stored-data vllm_buildkite_builds.json
python dynamo2ch.py --dynamodb-table vllm-buildkite-job-events --clickhouse-table fortesting.vllm_buildkite_jobs --stored-data vllm_buildkite_jobs.json

@huydhn huydhn requested a review from clee2000 August 13, 2025 04:13
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 13, 2025
Copy link

vercel bot commented Aug 13, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Project Deployment Preview Updated (UTC)
torchci ⬜️ Ignored Aug 13, 2025 4:13am

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants