Use Travis CI to download Rubygems data for listed gems

[ashmaroli]

The following Rake task takes the directory listing (_data/themes.yml and _data/plugins.yml clubbed into a single _data/entries.yml) and downloads each gem into their respective directory

# frozen_string_literal: true

require "rubygems"
require "rake"
require "open-uri"
require "safe_yaml"

task :default do
  entries = SafeYAML.load_file("_data/entries.yml")
  entries.each do |type, list|
    FileUtils.mkdir_p("_data/#{type}")
    list.each do |item|
      begin
        data = open("https://rubygems.org/api/v1/gems/#{item}.yaml").read
        File.open("_data/#{type}/#{item}.yml", "wb") { |file| file.puts data }
      rescue OpenURI::HTTPError
        puts "#{item} does not seem to exist at rubygems.org"
        next
      end
    end
  end
end

The data files can then be used to define the front end..
We can then set up Travis to download these files at a set interval..
Thoughts?

[mattr-]

How would _data/entries.yml get created?

Also, in order for this to work, Travis would have to commit these back to GitHub, would it not? I'm not sure that's something they would support doing. I'm also not sure if I've understood your idea correctly. Let me know if I've got it wrong.

[ashmaroli]

How would _data/entries.yml get created?

If this idea gets approved, I'll open a PR that clubs existing _data/themes.yml and _data/plugins.yml files into _data/entries.yml and add the rake task to this repo as well. In future, any user desiring to get their plugin listed need to open a PR adding their gem / plugin-name / theme-name to _data/entries.yml

Travis would have to commit these back to GitHub, would it not?

Yes, Travis CI can be used to do this.. Its documented. My plan is therefore:

• set up Travis to commit back to this repo. (I've already run trials on my fork). Perhaps @parkr can lend a hand at adding a secure access token for this repo..)
• instruct Travis to:
  • download gem data from Rubygems.org API if available
  • use this data to build the production-ready site
  • push the generated site back to gh-pages branch on this repo (default)
  • set up cron jobs to have this automated at set intervals
  • update Readme document here to ease contribution and development

[mattr-]

I'm onboard with this. 👍

Out of curiousity, what drove the idea behind creating a single file vs. keeping the two files and just smashing them together as part of the rake task?

[DirtyF]

👍 Interesting, right now we have manually added documents, this would allow us to display more info like latest version, author, downloads, link to changelog, etc. 😄

Downloads could be use as a sort filter.

{
    "authors": "Mark Otto",
    "bug_tracker_uri": null,
    "changelog_uri": null,
        "runtime": [
            {
                "name": "jekyll",
                "requirements": "~> 3.6"
            },
            {
                "name": "jekyll-feed",
                "requirements": "~> 0.9"
            },
            {
                "name": "jekyll-gist",
                "requirements": "~> 1.5"
            },
            {
                "name": "jekyll-paginate",
                "requirements": "~> 1.1"
            },
            {
                "name": "jekyll-sitemap",
                "requirements": "~> 1.1"
            }
        ]
    },
    "documentation_uri": "http://www.rubydoc.info/gems/jekyll-theme-poole/2.1.0",
    "downloads": 857,
    "gem_uri": "https://rubygems.org/gems/jekyll-theme-poole-2.1.0.gem",
    "homepage_uri": "https://github.com/poole/poole",
    "info": "The Jekyll Butler, designed and developed by @mdo to provide a clear and concise foundational setup for any Jekyll site.",
    "licenses": [
        "MIT"
    ],
    "metadata": {
        "plugin_type": "theme"
    },
    "name": "jekyll-theme-poole",
    "project_uri": "https://rubygems.org/gems/jekyll-theme-poole",
    "version": "2.1.0",
 
}

[ashmaroli]

what drove the idea behind creating a single file vs. keeping the two files and just smashing them together as part of the rake task?

Easier to code to load a single YAML file and proceed.., as against reading two files and merging data or clobbering the two files into one and then loading that.... 😄

[ashmaroli]

allow us to display more info

yes.. though we still need the manual data in sync with showcasing themes and plugins that have not been gemmified..

[pathawks]

though we still need the manual data in sync with showcasing themes and plugins that have not been gemmified..

I think this directory should only include plugins and themes that are available as Gems

[pathawks]

No! This is not going to die due to inactivity!

What do we need to do to carry this across the finish line?

[pathawks]

What do we need to do to carry this across the finish line?

[pathawks]

Is there a PR for this?

[ashmaroli]

Nope. I had one somewhere around in Feb, but lost it to a corrupted Git index.
Now I don't remember what exactly I planned on doing at the time..

[parkr]

This should be fairly straightforward of a process: list all gems, hit the rubygems metadata endpoint for each gem (costly, do once per month?), write yaml, commit. I’d can get a deploy key for this repo from @jekyllbot, but want to make sure it can only be used by Travis’s scheduler instead of any person running a Travis build.