Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up sources.sh even more #118

Merged
merged 1 commit into from
Feb 28, 2025
Merged

Conversation

yosifkit
Copy link
Member

@yosifkit yosifkit commented Feb 28, 2025

by improving slow jq section in sources.sh (building upon the gains from #115)

Local speed improvement:

$ cp sources.json sources-copy.json
$ # before change
$ time ../meta-scripts/sources.sh --cache-file=sources-copy.json > sources.json

real    1m15.459s
user    1m14.891s
sys     0m0.324s
$ # after
$ time ../meta-scripts/sources.sh --cache-file=sources-copy.json > sources.json

real    0m15.600s
user    0m15.071s
sys     0m0.477s

$ # with `sources.json` being correctly generated and unchanged from the checkout.

After tianon's improvements and a reorder 😍

$ time ../meta-scripts/sources.sh --cache-file=sources-copy.json > sources.json

real    0m5.095s
user    0m4.570s
sys     0m0.277s

And on a Jenkins worker versus the ~7 minutes in #115 (pre-feedback changes): 🎉 🎊 🥳

$ time ../doi-meta-scripts/sources.sh --cache-file=sources-copy.json > sources-cached.json

real    0m44.134s
user    0m43.897s
sys     0m1.458s

@yosifkit yosifkit requested a review from a team as a code owner February 28, 2025 22:08
sources.sh Outdated
Comment on lines 279 to 281
value: {
($e.arches | keys[]): [$e.sourceId]
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
value: {
($e.arches | keys[]): [$e.sourceId]
}
value: { ($e.arches | keys[]): [$e.sourceId] },

tianon
tianon previously approved these changes Feb 28, 2025
sources.sh Outdated
| ( $s.arches[] | .tags[], .archTags[] )
| {
key: .,
value: { ($s.arches | keys[]): [$s.sourceId] }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
value: { ($s.arches | keys[]): [$s.sourceId] }
value: { ($s.arches | keys[]): [$s.sourceId] },

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should also add a comment that this happens pre-arches-merge so looping over arches twice is fine?

by improving slow jq section in sources.sh
@tianon tianon merged commit 3e3f75b into docker-library:main Feb 28, 2025
1 check passed
@tianon tianon deleted the moar-speed branch February 28, 2025 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants