Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is this gem maintained? #490

Open
bloudermilk opened this issue Apr 19, 2018 · 32 comments
Open

Is this gem maintained? #490

bloudermilk opened this issue Apr 19, 2018 · 32 comments

Comments

@bloudermilk
Copy link

Our team is evaluating some options to read from our follower DBs on Heroku for select analytics workloads. We found this gem and while it seems to handle our use case, we had some issues in practice, which led us to the Issues tab.

It seems that issues and PRs are piling up over the last few of months, leaving us worried that we're adopting a library that might no longer be maintained. Of course we're very grateful to the OSS authors who shared this library with the public, but practically speaking it wouldn't be responsible for us to use this gem if it's been abandoned.

So I must ask, without any judgement, do the authors intend to continue maintaining this library?

@thiagopradi
Copy link
Owner

hey @bloudermilk - I try to do my best to work on the open issues / requests, but since I'm not using the library on a production anymore, It's been pretty hard to allocate time to work on it.

That said, there are a few users who are using the library successufully in their production environments. I would recommend you to try it out for a few days and see if it fits your use case.

Hopefully my comment was helpful.

Thiago

@pboling
Copy link

pboling commented Apr 19, 2018

@bloudermilk From the perspective of a taker:

  1. Many people use this library, and, IMO, it needs to be passed to community maintenance.

  2. I use one feature of this gem, to direct queries explicitly to a master or a follower, and for that it usually works. There are a number of quirks and edge cases, as you will see in /issues. But I use that one feature on an absolute monster of a system (intricately.com - billions of jobs per month, trillions of queries, terabytes of PG data, hundreds of millions of records), and it has worked without any significant issues (nothing that didn't have a simple workaround).

  3. Using the gem is not transparent. Crumbs of this gem will snake through the code.
    In certain configurations, like my setup, you do have to explicitly set the current_shard on objects before saving them, when they originate on the follower, for example. So you do need to commit to it when you choose it.

  4. Judging from what @thiagopradi himself has said on other threads, the internals of the gem are in need of refactoring to solve some of the trickier things.

  5. It looks like Rails 5.2 support may be difficult, or may necessitate that refactoring. For this project to survive there needs to be a community effort. @thiagopradi have you considered creating a Github Org, or adding other maintainers/owners to your repo?

  6. I evaluated the landscape of tools like this for another project recently, and this project is still the best option, IMO, despite all the risks 🌟

This is a great discussion to have! I hope more people will jump in here, and I hope some of those will offer support. I just took over the oauth2 project which was in a similar problematic state of near-death, and don't have any extra bandwidth or I would volunteer.

@erkie
Copy link
Contributor

erkie commented Apr 20, 2018

@thiagopradi thanks for jumping in and responding here. I'm just curious how we can get Rails 5.2 support merged/worked on. The PR I made worked in production for us, but there are certain aspects of it that I think could be improved. So a discussion with people in-the-know would be great to have. (Please don't take it as me trying to pressure you or anyone to take a look at the PR! I'm just curious what the normal way of going about would be)

Are you the only maintainer on this repo with merge access? If not, how/where do you communicate/congregate? I definitely think a community org is a great idea to let this gem thrive!

@bloudermilk
Copy link
Author

Thanks everyone for the responses so far–your promptness and thoughtfulness do well to answer my original question.

@thiagopradi I know first hand the reality of past OSS work becoming personally irrelevant and hard to dedicate time to as a result. I've had to leave projects behind in the past, but I wasn't nearly as successful in creating something popular in the first place. I think the continued popularity of this gem speaks to its utility and opportunity to outlive its original purpose at your previous org. Perhaps the suggestions above are reasonable and there are a few people with sufficient knowledge of the codebase to help maintain/refactor it?

@pboling thank you for your perspective and vote of confidence. It sounds like we have very similar or identical use-cases, which is encouraging. We're still on Rails 5.1 so it seems there is hope that we can make it work. I will spend a few more hours this week tinkering to see if that's the case.

@JasonLunn
Copy link

How many open issues are there in total impacting Rails 5.x support? I can think of:

5.2:
#477
#489

5.1+:
#452

5.0+:
#478
#467

Have I missed any?

@thiagopradi
Copy link
Owner

Hi @bloudermilk, @pboling, and @erkie - thanks for the insights and comments on this thread. I'm going to try to summarize my thoughts about the mentioned topics here:

1 - Regarding passing it along to community maintenance, I agree and I'm willing to do this. However, I've never found someone who could contribute regularly to the Gem, working on the code and answering the issues. If any of you guys want to be a maintainer, please ping me.

2 - Regarding the fact that the Gem is not transparent, I agree 100%. I coded this Gem almost eight years ago in the context of Rails 2.x - which was super coupled and pretty hard to extend - which lead to a lot of monkey patch on the internals of the Gem. The Gem does need a big refactor to be more reliable, mainly when using in big legacy projects.

3 - On the matter of the Rails 5.x support, I'll try to allocate some time to work on it this week. I'll try to merge all the fixes and get a release that is compatible with Rails 5.x.

I'll keep this thread open for discussing regarding moving the gem to community maintenance - if you guys have any tips on this subject, please let me know.

Thiago

@JasonLunn
Copy link

@thiagopradi - what's the state of your efforts? Is there anything that can be done to help?

@pboling
Copy link

pboling commented May 11, 2018

@JasonLunn I think most critically we need people with available time to work on the issues noted here: #490 (comment) (three comments above this one)

@JasonLunn
Copy link

I don't think any one wants to take on any of the above without hearing from @thiagopradi first so as not to create redundant effort.

@adomokos
Copy link
Contributor

Would it simplify things if this gem was refactored to support only AR 5.x+ and released under 2.0 version with no backward compatibility? We use it in pure Ruby apps with only AR dependency in them.

The reason I write this is to decrease the scope of this proposed refactoring.

@JasonLunn
Copy link

I'd have no objection to any approach that didn't break JRuby support.

@pboling
Copy link

pboling commented May 16, 2018

This gem is struggling for maintenance with the current support load, so I vote to reduce the scope of supported Rubies and Railses with the next major release... Keeping latest JRuby 9k and MRI Ruby stable at minimum.

@lunaru
Copy link

lunaru commented Sep 28, 2018

@pboling you mentioned that you evaluated several other alternatives. We're using this gem at reamaze.com for the exact same as what you've mentioned (sending reads to primary vs secondary databases). Has your evaluation of the landscape changed in the last half-year since your original comment? It's an interesting conundrum if the primary sharding/read-splitting gem in the Rails community is effectively on life-support.

@pboling
Copy link

pboling commented Sep 29, 2018

@lunaru This is this the gem we are using on a hugely scaled application that is pushing the limits of PostgreSQL v9, having many tables with hundreds of millions of rows, supporting many thousands of connections, and simultaneous queries. I haven't re-evaluated the landscape at all since my comment above.

We are planning to migrate from PG 9.6 to 11 when it is released (soon!) and I expect to run into problems with this library as we will begin using the partitioning features of PG 11 as the next step in our search for ways to keep the data beast humming.

At that point I will either be forced to take an active interest in this project, or write/find an alternative.

UPDATE: We migrated to v10, and v11 without issues.

@pboling
Copy link

pboling commented Oct 15, 2018

Many, if not all, of the features of Octopus may be part of Rails 6!
rails/rails#34052

I am not yet sure how we will bridge the gap from Rails 4.2 to Rails 6 yet though. This gem may still play a role in a migration that steps through Rails 5.2, if we can get a release together that supports 5.2 well.

@thiagopradi
Copy link
Owner

Hi @pboling - I actually plan to create a migration guide (for Rails 6) from Octopus to the multi db API that is going to be released with Rails. For Rails 5.2 - I'm doing what I can to work on an Octopus release for Rails 5 - but I hadn't much free time in the last month.

@lunaru
Copy link

lunaru commented Oct 15, 2018

@thiagopradi I want to thank you for your continued involvement and updating us with the plan. It would be perfect if we could have a 5.2 compatible octopus to tide us over for the migration to Rails 6.x.

I think the reality is that even with 6.x on the horizon it will be months if not years for folks to fully migrate over so a 5.2 compatible octopus is still in heavy demand and much appreciated by the community. We have a codebase we're trying to migrate to 5.2 so if you need any help with testing your 5.2 branch, I'd be happy to provide feedback and help where I can from our team.

@kevinjcoleman
Copy link
Contributor

@thiagopradi not sure if you saw this PR #507, but I solved some of the migration issues you were having on the previous branch. I know you were looking for help previously, so I'm willing to jump in where ever you need. I'm testing that branch I created today in our Rails 5.2 app.

@thiagopradi
Copy link
Owner

@kevinjcoleman I haven't seen yet - I'm going to review it this week. It seems pretty feature-complete for am Octopus release that supports Rails 5.2 👏 👏 . Let me know if your tests with your app are successful.

@thiagopradi
Copy link
Owner

@kevinjcoleman - can you run your tests directly from my branch (feature/updating-octopus-versions) ? Tests are all green - I just want to make sure it works with a real Rails 5.2 app. Thanks!

@kevinjcoleman
Copy link
Contributor

kevinjcoleman commented Oct 16, 2018

@thiagopradi I've started testing. For the most part it works, however I'm running into an issue, I don't think this is rails 5.2 specific due to this issue #421, where running rake db:migrate is attempting to run migrations on my replicated slave and therefore breaking the replication due to duplicate key entry constraints in the ar_internal_metadata. It looks like those migrations are being run because connection.shards is being passed here rather than filtering by which shards are for that migration. https://github.com/thiagopradi/octopus/blob/feature/updating-octopus-versions/lib/octopus/migration.rb#L113

@kevinjcoleman
Copy link
Contributor

@thiagopradi ok the migration issues I was having are solved by this PR #508. If I'm understanding the way using is setup for migrations, we should only be passing the shards for a given migration like that.

@thiagopradi
Copy link
Owner

Hello everyone - an important announcement about Octopus: it will enter into maintenance mode once Rails 6 is released. Rails 6 will have all the features necessary for sharding built-in into Rails core: rails/rails#34052. I'll provide a migration guide for everyone that is using Octopus to upgrade to the native AR API.

@krzcho
Copy link

krzcho commented Jan 22, 2019

@thiagopradi any chance on migration guide as we have rails 6 beta available?

@thiagopradi
Copy link
Owner

@krzcho - I'll try to get it done for the Rails 6.0 release in April.

@dhempy
Copy link

dhempy commented Aug 29, 2019

I am implementing Octopus in our Rails 4.2 application for reading from a replica. Default is fully_replicated: false, and we're only .using(:reader_db) for a handful heavy queries for reports.

Simultaneously, another team member is working on upgrading our app from Rails 4.2 to Rails 5.

I only just now, reading this thread, considered that Octopus may have issues under Rails 5. Can anyone advise if that's a turnkey/easy/painful transition from Rails 4.2?

@kevinjcoleman
Copy link
Contributor

kevinjcoleman commented Aug 29, 2019

@dhempy honestly if you can I would wait until you get on Rails 6.1, it will have support for multiple databases and it will be under more consistent development than octopus. Doing that would be easier than trying to migrate from octopus to rails multiple databases at a later point which is a task I'm already dreading. https://edgeguides.rubyonrails.org/active_record_multiple_databases.html

@mobilutz
Copy link

@thiagopradi Is there any update on the migration guide?
Thanks for a reply.

@sdrioux
Copy link

sdrioux commented Oct 29, 2019

Hi @pboling - I actually plan to create a migration guide (for Rails 6) from Octopus to the multi db API that is going to be released with Rails. For Rails 5.2 - I'm doing what I can to work on an Octopus release for Rails 5 - but I hadn't much free time in the last month.

@thiagopradi Its been a year since this comment. Can you please write a migration guide or consider adding contributors to this gem?

@tibbon
Copy link

tibbon commented Oct 26, 2020

We're going to need to migrate soon, and even if there's no formal guide... any experience or hints about other people's migration experiences would be appreciated.

@veerae
Copy link

veerae commented Oct 6, 2023

Is there any migration guide available, we are migrating our application into rails6, I need migration guide. Please provide soon if any one have

@krzcho
Copy link

krzcho commented Oct 6, 2023

@veerae I have switched to octoball

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests