Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce disk space usage of the rules #2641

Open
luispadron opened this issue Jan 25, 2025 · 5 comments
Open

Reduce disk space usage of the rules #2641

luispadron opened this issue Jan 25, 2025 · 5 comments

Comments

@luispadron
Copy link
Contributor

This is a general issue to describe the current disk space usage issues of some of the rules.

The current default is to not use the use_tree_artifacts_output flag, this mostly seems to be because of potentially now fixed issues in Bazel like: bazelbuild/bazel#16361. See also: #2545

With the current default the disk space usage for a large-ish application target is >3x the size when compared to the tree artifact ouputs.

Without usage of the use_tree_artifacts_outputs flag (default):

bin/Code/Apps/App/App_archive-root/Payload/App.app (732 MB)
bin/Code/Apps/App/App-intermediates/unprocessed_archive.zip (732 MB)
bin/Code/Apps/App/App.ipa (740 MB)

With use_tree_artifacts_outputs flag:

bin/Code/Apps/App/App_archive-root/Payload/App.app (732 MB)

For large projects this can add several hundreds of gigabytes to the output-base which means engineers need to frequently bazel clean --expunge to clear up disk space (or worse run into failed builds when they run out of disk space).

This has also been a topic of concern for other users, see: #2638

@luispadron
Copy link
Contributor Author

luispadron commented Jan 25, 2025

To start, do we need unprocessed_archive.zip, at least going off this comment it seems related to Tulsi which is AFAIK deprecated:

# provider, is used by Tulsi for custom processing logic. (b/120221708)
unprocessed_archive = intermediates.file(
actions = actions,
target_name = rule_label.name,
output_discriminator = output_discriminator,
file_name = "unprocessed_archive.zip",
)

@brentleyjones
Copy link
Collaborator

I don't think we need it.

luispadron added a commit that referenced this issue Feb 10, 2025
This output, as the comment says, is intended for Tulsi which is now deprecated.
As part of #2641 we are removing this output to save on disk space utilization.
luispadron added a commit that referenced this issue Feb 10, 2025
This output, as the comment says, is intended for Tulsi which is now deprecated.
As part of #2641 we are removing this output to save on disk space utilization.
@luispadron
Copy link
Contributor Author

luispadron commented Feb 10, 2025

Turns out the comment saying this was for Tulsi only is incorrect, its the archive used when not using tree artifact outputs.

@keith
Copy link
Member

keith commented Feb 10, 2025

FWIW i think the unprocessed archive predates the quality tree artifact support in bazel in general. back in 2018 those just didn't work at all. so i imagine there is some way to eliminate it at this point.

@luispadron
Copy link
Contributor Author

Yeah id love to make tree artifacts the default, theres this blocker right now: #2545

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants