OS X Setup

Note: We strongly encourage using Docker for your development environment. Due to limited resources, we can't offer support for all local installs, so please proceed with the understanding that you'll have to do most of the troubleshooting on your own.

Table of Contents Setup and installation 1. Install a package manager 2. Get Ruby 3. Install, start, and configure prerequisites 4. Get the source code 5. Install the gems 6. Configure MySQL 7. Configure Redis 8. Load the database schema 9. Update your local config file 10. Run the app 11. Seed your database and load the site styles 12. Create Elasticsearch indexes 13. Get a nicer URL 14. Set up your test database Known Issues Install Nokogiri 1.6.8 - October 2016 MySQL sql_mode=only_full_group_by error MySQL utf8mb4 upgrade

Setup and installation

These instructions assume you are comfortable with your shell of choice. They also assume that you've installed Xcode and therefore have the GNU toolchain. None of this needs to be run as root.

1. Install a package manager

These instructions will assume you have installed the package manager Homebrew. To get Homebrew:

  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

2. Get Ruby

While macOS comes with Ruby pre-installed, you'll want to use the same version as us to avoid compatibility issues.

There are two competing ways of managing Ruby environments on Unixes these days: rbenv and RVM. If you're already using one of these, you can set it to Ruby 3.1.4 and you're done. Otherwise, you'll need to install one and get yourself rubied up.

These instructions will assume you're using rbenv. To install rbenv using Homebrew:

  brew install rbenv rbenv-gemset openssl

Follow the setup instructions on the rbenv page to get it into your shell environment. Then run rbenv to install Ruby 3.1.4:

  rbenv install 3.1.4
  rbenv global 3.1.4
  rbenv rehash
  ruby -v

If ruby -v does not return the Ruby version you have set, you may still need to add rbenv to bash.

3. Install, start, and configure prerequisites

 brew install git imagemagick redis [email protected]

Note: If you intend to allow downloads (ePub, PDF, etc) from your archive, you will need to add "calibre" to your package install list.

In the output will be two caveat blocks containing additional information on setting up Redis and MySQL. A version of that information is included here, but instructions can vary from version to version, so make sure to follow the directions from your console.

Follow the brew startup instructions for Redis by copying and pasting the desired command from the caveat.

 ==> Caveats  
 To have launchd start redis now and restart at login:
   brew services start redis
 Or, if you don't want/need a background service you can just run:
   /opt/homebrew/opt/redis/bin/redis-server /opt/homebrew/etc/redis.conf

Because we're on an alternate version of MySQL, we'll need to force a symlink before we can follow the other setup instructions:

  brew link [email protected] --force

Then run mysql_secure_installation and follow the instructions for setting up MySQL databases. MySQL is the most annoying of the packages you have to deal with here, but the brew instructions work. Make a note of the user name and password you've used to set up MySQL. You'll need it to configure your database.

 ==> Caveats
 We've installed your MySQL database without a root password. To secure it run:
   mysql_secure_installation
  
 MySQL is configured to only allow connections from localhost by default
  
 To connect run:
   mysql -uroot
  
 [email protected] is keg-only, which means it was not symlinked into /opt/homebrew,
 because this is an alternate version of another formula.
  
 If you need to have [email protected] first in your PATH, run:
   echo 'export PATH="/opt/homebrew/opt/[email protected]/bin:$PATH"' >> ~/.zshrc
  
 For compilers to find [email protected] you may need to set:
   export LDFLAGS="-L/opt/homebrew/opt/[email protected]/lib"
   export CPPFLAGS="-I/opt/homebrew/opt/[email protected]/include"
  
 For pkg-config to find [email protected] you may need to set:
   export PKG_CONFIG_PATH="/opt/homebrew/opt/[email protected]/lib/pkgconfig"

 To have launchd start [email protected] now and restart at login:
   brew services start [email protected]
 Or, if you don't want/need a background service you can just run:
   /opt/homebrew/opt/[email protected]/bin/mysqld_safe --datadir\=/opt/homebrew/var/mysql

You'll also need to install and start Elasticsearch. Unfortunately, Elasticsearch no longer supports Homebrew, so we'll need to take a different approach.

1. Download Elasticsearch 7.17.1 from the Elasticsearch website, unzip it, and drag the elasticsearch-7.17.1 folder somewhere easy to find, e.g., your Documents folder.
2. Download Java 8 from java.com and install it.
3. In the console, navigate to your Elasticsearch bin folder, e.g., cd Documents/elasticsearch-7.17.1/bin
4. Start Elasticsearch: ./elasticsearch
   • You might need to disable Gatekeeper if you run into errors like these.
   • You can set $JAVA_HOME if it complains about that.

4. Get the source code

Now you can clone the source:

  git clone https://github.com/otwcode/otwarchive.git 

If you want to participate in the project and create pull requests the easy way, you should probably make yourself an account here on GitHub if you don't already have one. Fork the repo! Get a copy of your repo locally using either the GitHub for Mac client or the command line:

  git clone [email protected]:YOURGITHUB/otwarchive.git
  git remote add upstream https://github.com/otwcode/otwarchive.git

You add the original repository as a remote so you can track changes made to official repo.

5. Install the gems

cd to the project directory. You're now looking at a typical Rails app. It uses Bundler to manage dependencies, so you can install those:

  gem install bundler
  rbenv rehash
  bundle install

If you find that bundle install fails when installing the Mysql2 gem, you may have installed a newer and incompatible version of MySQL, such as version 8.x. You can check your version with mysql -V.

If your MySQL version is correct, you may have to configure Bundler to use Homebrew's openssl directory when installing MySQL2.

6. Configure MySQL

  cp config/database.example config/database.yml

Edit the database.yml file and give it the MySQL username and password combo you used earlier. Test that your setup is working by running the rake task to build the database tables:

  bundle exec rake db:create:all

7. Configure Redis

  cp config/redis.example config/redis.yml

If you're not already using Redis in other projects, change dev.ao3.org to localhost and move to the next step. If you are, read on.

Unfortunately, the Redis initialization forces you to set up Redis using a hostname:port string instead of just taking a hash like you might expect. This doesn't let you specify which Redis database to use, which if you've already got data in a Redis instance is unacceptable. Let's fix that.

Edit config/initializers/gem-plugin_config/redis.rb. Replace these two lines:

  redis_host, redis_port = redis_config[rails_env].split(":")
  $redis = Redis.new(:host => redis_host, :port => redis_port)

With this one:

  $redis = Redis.new(redis_config[rails_env])

Put something like this into your redis.yml. Your db numbers might vary.

 test:
        :host: localhost
        :port: 6379
        :db: 17
 development:
        :host: localhost
        :port: 6379
        :db: 16
 production:
        :host: localhost
        :port: 6379
        :db: 16

8. Load the database schema

  bundle exec rake db:schema:load
  bundle exec rake db:migrate

Note: If you're getting an error on schema:load, you may need to comment out factory_girl and factory_girl_rails in the Gemfile, bundle, and try running bundle exec rake db:schema:load again. When you're done, don't forget to uncomment the lines and run bundle install again.

9. Update your local config file

Edit config/local.yml and add anything to it that you want to change from config/config.yml. The file must be present even if empty.

10. Run the app

  bundle exec rails server

If you're on Yosemite (OS X 10.10) or newer, you will need to use bundle exec rails server -d.

You might now need to fix a YAML engine problem. If Rails eventually exits with a yaml/psych parse error, edit config/boot.rb and add this line after Bundler.setup:

  YAML::ENGINE.yamler = 'syck'

Newer versions of Ruby are using a YAML engine that doesn't allow symbols in YAML files, and the OTW localization files are full of symbols. The syck engine is tolerant of symbols. Another way to solve this problem is to use rbenv to install an older Ruby. 1.9.0 and 1.9.1 should work, but will have bugs fixed in later versions of Ruby.

You can now visit http://localhost:3000/ in your browser of choice and see an empty local archive.

11. Seed your database and load the site styles

  bundle exec rake db:otwseed
  bundle exec rake work:missing_stat_counters
  bundle exec rake skins:load_site_skins

You may also wish to create additional admin and user accounts.

12. Create Elasticsearch indexes

To create your search indexes (and fill them with data), run the tasks in search.rake:

  bundle exec rake search:index_tags
  bundle exec rake search:index_works
  bundle exec rake search:index_pseuds
  bundle exec rake search:index_bookmarks

Note: If the above commands error with Failed to open TCP connection to 127.0.0.1:9400, you might need to reconfigure to the port your Elasticsearch is running on (i.e. Homebrew uses :9200 by default). Add the port ES_URL: "http://localhost:9200" to local.yml.

To make sure your indexes get updated (e.g., when you create a new work), you'll need to start up a Resque worker and make sure Resque Scheduler is running:

 RAILS_ENV=development QUEUE=* bundle exec rake environment resque:work
 bundle exec rake resque:scheduler

13. Get a nicer URL

Now that you know you have a working install, you can install Puma-dev and use prettier URLs like http://otwarchive.test/. Recommended for anybody working on more than one rack app project at a time.

14. Set up your test database

  bundle exec rake db:test:prepare

You'll need to edit redis-cucumber.conf to get Redis to work with cucumber tests. Remove the line that says

  dbfilename log/redis-cucumber-dump.rdb   

Known Issues

Install Nokogiri 1.6.8 - October 2016

If you're using a MacBook and it won't bundle install the nokogiri v.1.6.8 gem, you need to follow these instructions from http://john.hu/node/89:

Check your version of libxml2:

  brew list libxml2

If it is not installed and/or isn't libxml2 2.9.4, please use the following commands to upgrade it:

  brew update
  brew upgrade libxml2

After that, if you cannot install nokogiri 1.6.8 with the same error, you may also need to upgrade code-selected:

  xcode-select --install

Then run bundle install again.

MySQL sql_mode=only_full_group_by error

If you receive an error like the following:

  Mysql2::Error: Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'otwarchive_development.subscriptions.id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by: SELECT `subscriptions`.* FROM `subscriptions`  WHERE ((subscribable_id = 1677550 AND subscribable_type = 'Work') 
            OR (subscribable_id IN (202514) AND subscribable_type = 'User')
            OR (subscribable_id IN (NULL) AND subscribable_type = 'Series')) GROUP BY user_id

You may need to edit your MySQL config settings as described in Anuj Gakhar's "MySQL Error caused by sql_mode=only_full_group_by" blog post.

1. Copy the default config file:

  sudo cp $(brew --prefix mysql)/support-files/my-default.cnf /etc/my.cnf

If my-default.cnf is missing, try:

  sudo cp /usr/local/etc/my.cnf /etc/my.cnf

2. Edit the file to remove to remove ONLY_FULL_GROUP_BY and STRICT_TRANS_TABLES from sql_mode (you may only have one of these):

  sudo nano /etc/my.cnf

Example addition to my.cnf:

  sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

3. After updating my.cnf, restart your MySQL server.

MySQL utf8mb4 upgrade

In December 2018, we converted the Archive's database to utf8mb4 (AO3-4056). To convert your development database:

1. Edit your MySQL config file (see the previous section for its location) to have:

[mysqld]
innodb_file_format = Barracuda
innodb_large_prefix = 1
innodb_file_per_table = ON

Note: If your MySQL server is greater than 5.7.7, innodb_file_format is already set to Barracuda, and does not need to be set.

2. Restart your MySQL server:

brew services restart mysql

3. Update your config/database.yml. If you find a section like this:

encoding: utf8
collation: utf8_general_ci

change it to:

encoding: utf8mb4
collation: utf8mb4_unicode_ci

3. Create a new migration ConvertDatabaseToUtf8mb4, as seen in the guide "Converting a Rails database to Utf8mb4 without downtime or data loss". The migration has a bug (AO3-5611), so you need to replace both instances of column.default.blank? with column.default.nil?. Then run ConvertDatabaseToUtf8mb4.

4. Repeat the previous step for your test database. To check if your test database is converted correctly, you can run the test to create a work with emojis in its title and content.