Gomig

Synchronize data between databases, currently only supports syncing from MySQL to PostgreSQL as that's what I need at the moment. The architecture is loosely based on py-mysql2pgsql and I've tried to keep it extensible, so with some help it should support Postgres to MySQL, MySQL to Mysql, Oracle to Postgres et cetera. Pull requests welcome.

The default config file is called "config.yml", format is YAML, and the parameters are a strict superset of py-mysql2pgsql's configuration format. If no config file is present on the first run, the sample config file will be installed in its place.

Features

• Uses postgres' COPY FROM support for fast data transfers
• Define projections (views) in the source database so that they match a (reduced) form of tables in the destination database. Gomig will sync the data.
• Can execute SQL directly on the destination server or output to a file, just like py-mysql2pgsql.
• Will ROLLBACK when something goes wrong, leaving the destination database intact. The source database is never INSERT/UPDATE/DELETE'ed, only views or projection tables are created on request, they can be safely dropped should they somehow survive culling.

Usage

If you have the go runtime installed, then it's as easy as:

# get it!
$ go get github.com/aktau/gomig

# run it (assuming $GOPATH is in your $PATH, as I assume it would be for
# most Go devs, otherwise you'll have to cd to $GOPATH/bin)

# check out the options
$ gomig --help

# which prints something similar to:
# Usage:
#   go-app [OPTIONS]
#
# Application Options:
#   -v, --verbose  Verbose output
#
# Help Options:
#   -h, --help     Show this help message
#
# Available commands:
#   generate-config  Generate a sample config file in the current directory
#   migrate          Migrate data from a source database to a destination file/database
#   test             Test if a connection to the source and destination databases can be established
#   version          Print the version and supported backends

# generate a config file, edit it, then run
$ gomig generate-config
$ edit config.yml
# you can test if the connection succeeds if you want:
$ gomig test
# or you can go straight to running a migration
$ gomig migrate
# alternatively you can explicitly supply a config file:
$ gomig migrate -f config.yml

To update to the newest version later, you can just do:

$ go get -u github.com/aktau/gomig

Of course, with Go it's quite easy to just build binaries so that everyone can run it without needing anything but the standard C runtime (which is basically installed everywhere). But I'm not doing that just yet. I might do that if there's interest though.

Build requirements

• Go (>=) 1.2 (uses positional notation in fmt.Sprintf)

Used libraries & software

Package	Description	License
github.com/lib/pq	Go database driver for postgres	MIT
github.com/go-sql-driver/mysql	Go database driver for MySQL	MPL v2
github.com/jessevdk/go-flags	Go package for cmdline flag parsing	BSD
launchpad.net/goyaml	Go package for parsing/writing YAML	LGPL v3

Todo

• Convert more datatypes, and do it more accurately. Gomig only handles varchar, text, blob (binary), boolean, integer and float at the moment. Dates, times and timestamps are implemented without testing at the moment, so anyone who's interested should try it out.
• Possibly faster data migration with goroutines, as explained in this article. Would need to make things quite a bit more threadsafe for that though, or keep the goroutines internal to the bulk methods...
• Testing! There are no tests yet, which is a shame.
• Travis, when the tests are made, it would be nice to have Travis automatically run them on each commit.

Screenshot

Everybody loves screenshots, even of console apps.

License (BSD)

Copyright (c) 2013, Nicolas Hillegeer All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

• Neither the name of the {organization} nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.