Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merging homebrew-core and linuxbrew-core #7028

Closed
11 tasks done
MikeMcQuaid opened this issue Feb 12, 2020 · 30 comments
Closed
11 tasks done

Merging homebrew-core and linuxbrew-core #7028

MikeMcQuaid opened this issue Feb 12, 2020 · 30 comments
Labels
help wanted We want help addressing this linux outdated PR was locked due to age

Comments

@MikeMcQuaid
Copy link
Member

MikeMcQuaid commented Feb 12, 2020

Eventually we may want to merge homebrew-core and linuxbrew-core. Even if not, minimising the diff between these repositories makes life easier for the linuxbrew-core maintainers.

To do this will require a few things (not in any particular order):

Notes

@iMichka
Copy link
Member

iMichka commented Apr 13, 2020

@sjackman I would like to make gcc@5 the main gcc formula on linux.
brew install gcc would then install gcc 9, like it does on mac. This would allow to cleanup the gcc formula on linux, and ease the transition. Do you think this is doable?

@issyl0
Copy link
Member

issyl0 commented Apr 13, 2020

As of #7280 we've got an audit for legitimate uses_from_macos usage.

@sjackman
Copy link
Member

@sjackman I would like to make gcc@5 the main gcc formula on linux.
brew install gcc would then install gcc 9, like it does on mac. This would allow to cleanup the gcc formula on linux, and ease the transition. Do you think this is doable?

My intuition is that on Linux gcc should be an alias to gcc@5.

@sjackman
Copy link
Member

As of #7280 we've got an audit for legitimate uses_from_macos usage.

Thanks, Issy!

@iMichka
Copy link
Member

iMichka commented Apr 13, 2020

My intuition is that on Linux gcc should be an alias to gcc@5

This means that we need os-dependent aliases. Not sure I like that idea. And I still think it is surprising to brew install gcc and get something different than on mac.

@MikeMcQuaid
Copy link
Member Author

I'm with @iMichka that I think the cleanest solution is probably to make the gcc formula the same on both OSs. It's easier to have Homebrew/brew compiler selection code pick a different compiler when needed than it's going to be to have homebrew-core have different logic for GCC versions with the same name.

@sjackman
Copy link
Member

Makes sense. I'm on board.

@fdr

This comment has been minimized.

@sjackman

This comment has been minimized.

@fmeyer

This comment has been minimized.

@iMichka

This comment has been minimized.

@sjackman

This comment has been minimized.

@fdr

This comment has been minimized.

@sjackman

This comment has been minimized.

@fdr

This comment has been minimized.

@iMichka

This comment has been minimized.

@iMichka
Copy link
Member

iMichka commented Nov 10, 2020

Short status update, as some huge progress has been made lately:

  • I update the checklist on the top of this issue
  • Most of the important xorg formulae have been migrated to homebrew-core (which had the nice side-effect that some of maintainers started to work on getting rid of the x11 requirement :)). Around 13 remaining xorg formulae need to be migrated.
  • 13 linux-only formulae (out of 43) have been added to homebrew-core and have linux CI there.
  • When I started reducing the diff between both repos, there were more than 2000 OS.mac? statments. We are now below 1400.

@iMichka
Copy link
Member

iMichka commented Nov 10, 2020

There was a proposal from @SMillerDev to start adding linux bottles and more CI to homebrew-core as the migration is ongoing. Initially I was thinking to add all the bottles in one step at the end of the migration; but we can also take a step-by-step approach. We can not re-use linuxbrew-core bottles as-is, as the revisions are sometimes different there.

  • Where do we store the new linux bottles on bintray for homebrew-core? I guess in https://bintray.com/homebrew, and we keep the old linux bottles in https://bintray.com/linuxbrew?
  • I think we do not need to adapt the merge script from homebrew-core to linuxbrew-core, as the linux sha256 line will conflict anyway, so this is a good thing.
  • We will do double bottle builds for some time, in homebrew-core and then in linuxbrew-core. I hope that's fine
  • Remains the final question: how will the transition happen from linuxbrew-core to homebrew-core for our users, knowing that revisions might be different between both repos. We do not need to answer this question right now.
  • What is the impact of the migration on the shelf-life of old linux bottles? Is this still a thing? I know that @sjackman always thought it was important, but I personally do not have any need for this.

Thoughts?

@SMillerDev
Copy link
Member

I think the best option to have would be:

We bottle the formula that are explicitly enabled to be tested in homebrew-core and use those bottles in linuxbrew where we can having less and less bottles in the linuxbrew bintray.

Second best option:

We bottle the formula that are explicitly enabled to be tested in homebrew-core and rebuild the bottles in linuxbrew. Until we flip the switch and we disable linuxbrew.

Worst option:

We don't bottle in homebrew-core, but we keep the same flow until all formulae are upstreamed from linuxbrew and then we start bottles in homebrew-core.

@dawidd6
Copy link
Member

dawidd6 commented Nov 10, 2020

The first option looks great, but how can we know on which bintray account the Linux bottle actually is? This will become irrelevant when we merge the core taps together, but for now I think it would be a problem that needs to be sorted out first.

@sjackman
Copy link
Member

We bottle the formula that are explicitly enabled to be tested in homebrew-core and rebuild the bottles in linuxbrew. Until we flip the switch and we disable linuxbrew.

I like this option.

@SMillerDev
Copy link
Member

The first option looks great, but how can we know on which bintray account the Linux bottle actually is? This will become irrelevant when we merge the core taps together, but for now I think it would be a problem that needs to be sorted out first.

I'd imagine we can try the main one and fall back if it fails?

@iMichka
Copy link
Member

iMichka commented Nov 10, 2020

I would go for second option. It's less work, and probably easier to handle.

We just need to think about the final migration when we flip the switch, but that's all.

@MikeMcQuaid
Copy link
Member Author

Yup 👍🏻

  • We will do double bottle builds for some time, in homebrew-core and then in linuxbrew-core. I hope that's fine

Fine with me.

  • Remains the final question: how will the transition happen from linuxbrew-core to homebrew-core for our users, knowing that revisions might be different between both repos. We do not need to answer this question right now.

Will likely be some custom brew update logic, eventually. I can handle that when the time comes.

  • What is the impact of the migration on the shelf-life of old linux bottles? Is this still a thing? I know that @sjackman always thought it was important, but I personally do not have any need for this.

Personally: I don't think it's worth worrying about this. We should probably avoid deleting these bottles but I don't see any need to write code to make them work better.

@MikeMcQuaid
Copy link
Member Author

Oh and: great work on this @iMichka!

@iMichka
Copy link
Member

iMichka commented Nov 16, 2020

Ok. I'll work on setting up linux bottling in homebrew-core soon for a few main formulae. I think the same rule will apply to homebrew-core as linuxbrew-core: if a formula has a linux bottle, any revision bump or version bump should build a new bottle. If there is no bottle, nothing happens (no linux CI) to that formula.

I'll start working on this once I am done with the self-hosted runners project, unless someone wants to pick this up earlier.

@iMichka
Copy link
Member

iMichka commented Dec 23, 2020

The xorg migration is done!

@iMichka
Copy link
Member

iMichka commented Jan 28, 2021

Another step done: cellar information for specific platforms in bottle blocks is now merged!

@iMichka
Copy link
Member

iMichka commented Oct 31, 2021

All done here. Users were migrated to homebrew-core with release 3.3.0.

There is still an issue open at Homebrew/formulae.brew.sh#566 but it is being worked on.

Closing here.

@iMichka iMichka closed this as completed Oct 31, 2021
@MikeMcQuaid
Copy link
Member Author

Congratulations on the completion of this huge effort. Great work @iMichka and all.

@github-actions github-actions bot added the outdated PR was locked due to age label Dec 2, 2021
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 2, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
help wanted We want help addressing this linux outdated PR was locked due to age
Projects
None yet
Development

No branches or pull requests

8 participants