Skip to content

Commit

Permalink
Initial docs (#33)
Browse files Browse the repository at this point in the history
  • Loading branch information
nickschuch authored Jan 21, 2025
1 parent 0f78403 commit 1936f57
Show file tree
Hide file tree
Showing 32 changed files with 20,119 additions and 100 deletions.
53 changes: 53 additions & 0 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: Deploy Docusaurus to Pages

on:
# Runs on pushes targeting the default branch
push:
branches: [docs]

# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: "pages"
cancel-in-progress: true

jobs:
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}

runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v3

- name: Set up Node.js
uses: actions/setup-node@v3
with:
node-version: 20.x

- name: Install and Build
run: |
cd docs
npm install --non-interactive
npm run build
- name: Setup Pages
uses: actions/configure-pages@v3

- name: Upload artifact
uses: actions/upload-pages-artifact@v2
with:
path: docs/build

- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v2
108 changes: 8 additions & 100 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,110 +1,18 @@
MTK (MySQL Toolkit)
===================

A toolkit for exporting, sanitizing and packaging MySQL database.

## The Goal of this Project

To provide developers with the tools to share MySQL database dumps in a safe and repeatable manner.

## Example

The following example scenario will showcase how developers can use mtk to dump and package a MySQL image database.

### Dump a Sanitized Database with MTK

**Configuration File**

Prior to packaging, developers should assess the data that is stored within the database and determine what data should be sanitized or dropped.

The following mtk configuration file cover some of common Drupal 7/8 data which should be sanitized or dropped.

```bash
$ cat mtk.yml

---
rewrite:
# Drupal 8.
users_field_data:
mail: concat(uid, "@localhost")
# Quoting here denotes an explicit string rather than mysql expression.
pass: '"password"'
# Drupal 7.
users:
mail: concat(uid, "@localhost")
pass: '"password"'

nodata:
- cache*
- captcha_sessions
- history
- flood
- batch
- queue
- sessions
- semaphore
- search_api_task
- search_dataset
- search_index
- search_total

ignore:
- __ACQUIA_MONITORING__
```

**Dump the database**
A tool designed to guide developers in the right direction for identifying and resolving performance issues.

Now that you have a configuration file, it is time to dump the database.
<img src="/logo/mtk.png" width="100">

The following command will dump a MySQL database using the configuration file created in the previous step.
----

```bash
$ export MTK_HOSTNAME=127.0.0.1
$ export MTK_USERNAME=root
$ export MTK_PASSWORD=password
$ export MTK_CONFIG=mtk.yml

$ mtk-dump test > db.sql
```

### Build a Database Image using Docker

Next we can build a database image using the database dump created by the step prior.

First, create a Dockerfile which will import the sanitized database dump.

In this example we are using MySQL images from our [image repository](https://github.com/skpr/image-mysql).

```dockerfile
FROM docker.io/skpr/mysql:8.x-v3-latest

ADD db.sql /tmp/db.sql
RUN database-import local local local /tmp/db.sql
```

Next, build the image with Docker.

```bash
docker build -t docker.io/my/database:latest .
```

Hooray! You have successfully packaged a sanitized MySQL database image!

### Integrate

The database image can then be integrated into local development workflows using Docker Compose (or similar).

Below is an example of how this can be configured.
A toolkit for exporting, sanitizing and packaging MySQL database.

```bash
$ cat docker-compose.yml
## The Goal of this Project

---
version: "3"
To provide developers with the tools to share MySQL database dumps in a safe and repeatable manner.

services:
## Getting Started

# Services used as part of the local development environment.
mysql:
image: docker.io/my/database:latest
```
See our [documentation site](https://mtk.skpr.io) for a tutorial on how to get started.
20 changes: 20 additions & 0 deletions docs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Dependencies
/node_modules

# Production
/build

# Generated files
.docusaurus
.cache-loader

# Misc
.DS_Store
.env.local
.env.development.local
.env.test.local
.env.production.local

npm-debug.log*
yarn-debug.log*
yarn-error.log*
41 changes: 41 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Website

This website is built using [Docusaurus](https://docusaurus.io/), a modern static website generator.

### Installation

```
$ yarn
```

### Local Development

```
$ yarn start
```

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

### Build

```
$ yarn build
```

This command generates static content into the `build` directory and can be served using any static contents hosting service.

### Deployment

Using SSH:

```
$ USE_SSH=true yarn deploy
```

Not using SSH:

```
$ GIT_USER=<Your GitHub username> yarn deploy
```

If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.
9 changes: 9 additions & 0 deletions docs/blog/authors.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
nickschuch:
name: Nick Schuch
title: Lead Architect
url: https://github.com/nickschuch
image_url: https://github.com/nickschuch.png
page: true
socials:
bluesky: nickschuch
github: nickschuch
12 changes: 12 additions & 0 deletions docs/blog/containerized-databases-for-developers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
slug: containerized-databases-for-developers
title: Containerized Databases for Developers
date: 2018-07-07
authors: [nickschuch]
---

The first blog post to introduce the term database images back in 2018, revolutionizing the way databases are packaged and shared with development teams and CI/CD pipelines.

This blog post highlights how implementing MySQL Toolkit has enhanced and streamlined our development workflow.

https://www.previousnext.com.au/blog/containerized-databases-developers
12 changes: 12 additions & 0 deletions docs/blog/drupalsouth-2019.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
slug: drupalsouth-2019
title: Containerized Databases for Developers
date: 2018-12-04
authors: [nickschuch]
---

What would you do if your laptop was stolen? Have I got your attention? Yes? Good :) Now think about all the un-sanitised databases that were on that laptop.

Lets talk about packaging those up into sanitised, developer consumable artefacts.

[![Screenshot of Drupal South Talk](https://img.youtube.com/vi/W7lE1H-OCRs/0.jpg)](https://www.youtube.com/watch?v=W7lE1H-OCRs)
11 changes: 11 additions & 0 deletions docs/blog/patterns-local-development-nirvana.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
slug: patterns-local-development-nirvana
title: Patterns for Local Development Nirvana
date: 2024-04-24
authors: [nickschuch]
---

Recorded at DrupalSouth 2024 in Sydney, my session is a deep dive into our journey towards local dev environment nirvana and highlights
how database images levelled up our local development workflows.

https://www.previousnext.com.au/blog/containerized-databases-developers
55 changes: 55 additions & 0 deletions docs/docs/Integrations/aws-rds.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
sidebar_position: 3
---

# AWS RDS

MySQL Toolkit offers native integration with AWS RDS, enabling developers to seamlessly copy data from an Aurora MySQL-Compatible DB directly into Amazon S3. This can be achieved (under the hood) using the [SELECT INTO OUTFILE S3 statement](https://docs.aws.amazon.com/prescriptive-guidance/latest/archiving-mysql-data/export-data-by-using-select-into-outfile-s3.html).

## Data flow

This feature results in a backup and restore workflow that looks like the below.

### Backup

```mermaid
graph LR;
MTK --> RDS
RDS -- Store Table Structure --> SQL
RDS -- Export Data --> S3
```

### Restore

```mermaid
graph LR;
SQL -- Table Structure --> RDS
S3 -- Import Data --> RDS
```

## Performance

By utilizing this approach, we've observed a 30% improvement in performance for database backups and restores on [our hosting platform](https://skpr.com.au).

## Usage

Below demonstrates how the `mtk dump` command can be updated to enable AWS RDS integration.

**Before**

```bash
mtk dump -u USERNAME -pPASSWORD -h HOSTNAME DATABASE > export.sql
```

**After**

```bash
# Flags
mtk dump --provider=rds --rds-region=ap-southeast-2 --rds-s3-uri=s3://my-s3-bucket -u USERNAME -pPASSWORD -h HOSTNAME DATABASE > export.sql

# Environment variables
export MTK_PROVIDER=rds
export MTK_RDS_REGION=ap-southeast-2
export MTK_RDS_S3_URI=s3://my-s3-bucket
mtk dump -u USERNAME -pPASSWORD -h HOSTNAME DATABASE > export.sql
```
Loading

0 comments on commit 1936f57

Please sign in to comment.