-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
0f78403
commit 1936f57
Showing
32 changed files
with
20,119 additions
and
100 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
name: Deploy Docusaurus to Pages | ||
|
||
on: | ||
# Runs on pushes targeting the default branch | ||
push: | ||
branches: [docs] | ||
|
||
# Allows you to run this workflow manually from the Actions tab | ||
workflow_dispatch: | ||
|
||
permissions: | ||
contents: read | ||
pages: write | ||
id-token: write | ||
|
||
concurrency: | ||
group: "pages" | ||
cancel-in-progress: true | ||
|
||
jobs: | ||
deploy: | ||
environment: | ||
name: github-pages | ||
url: ${{ steps.deployment.outputs.page_url }} | ||
|
||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v3 | ||
|
||
- name: Set up Node.js | ||
uses: actions/setup-node@v3 | ||
with: | ||
node-version: 20.x | ||
|
||
- name: Install and Build | ||
run: | | ||
cd docs | ||
npm install --non-interactive | ||
npm run build | ||
- name: Setup Pages | ||
uses: actions/configure-pages@v3 | ||
|
||
- name: Upload artifact | ||
uses: actions/upload-pages-artifact@v2 | ||
with: | ||
path: docs/build | ||
|
||
- name: Deploy to GitHub Pages | ||
id: deployment | ||
uses: actions/deploy-pages@v2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,110 +1,18 @@ | ||
MTK (MySQL Toolkit) | ||
=================== | ||
|
||
A toolkit for exporting, sanitizing and packaging MySQL database. | ||
|
||
## The Goal of this Project | ||
|
||
To provide developers with the tools to share MySQL database dumps in a safe and repeatable manner. | ||
|
||
## Example | ||
|
||
The following example scenario will showcase how developers can use mtk to dump and package a MySQL image database. | ||
|
||
### Dump a Sanitized Database with MTK | ||
|
||
**Configuration File** | ||
|
||
Prior to packaging, developers should assess the data that is stored within the database and determine what data should be sanitized or dropped. | ||
|
||
The following mtk configuration file cover some of common Drupal 7/8 data which should be sanitized or dropped. | ||
|
||
```bash | ||
$ cat mtk.yml | ||
|
||
--- | ||
rewrite: | ||
# Drupal 8. | ||
users_field_data: | ||
mail: concat(uid, "@localhost") | ||
# Quoting here denotes an explicit string rather than mysql expression. | ||
pass: '"password"' | ||
# Drupal 7. | ||
users: | ||
mail: concat(uid, "@localhost") | ||
pass: '"password"' | ||
|
||
nodata: | ||
- cache* | ||
- captcha_sessions | ||
- history | ||
- flood | ||
- batch | ||
- queue | ||
- sessions | ||
- semaphore | ||
- search_api_task | ||
- search_dataset | ||
- search_index | ||
- search_total | ||
|
||
ignore: | ||
- __ACQUIA_MONITORING__ | ||
``` | ||
|
||
**Dump the database** | ||
A tool designed to guide developers in the right direction for identifying and resolving performance issues. | ||
|
||
Now that you have a configuration file, it is time to dump the database. | ||
<img src="/logo/mtk.png" width="100"> | ||
|
||
The following command will dump a MySQL database using the configuration file created in the previous step. | ||
---- | ||
|
||
```bash | ||
$ export MTK_HOSTNAME=127.0.0.1 | ||
$ export MTK_USERNAME=root | ||
$ export MTK_PASSWORD=password | ||
$ export MTK_CONFIG=mtk.yml | ||
|
||
$ mtk-dump test > db.sql | ||
``` | ||
|
||
### Build a Database Image using Docker | ||
|
||
Next we can build a database image using the database dump created by the step prior. | ||
|
||
First, create a Dockerfile which will import the sanitized database dump. | ||
|
||
In this example we are using MySQL images from our [image repository](https://github.com/skpr/image-mysql). | ||
|
||
```dockerfile | ||
FROM docker.io/skpr/mysql:8.x-v3-latest | ||
|
||
ADD db.sql /tmp/db.sql | ||
RUN database-import local local local /tmp/db.sql | ||
``` | ||
|
||
Next, build the image with Docker. | ||
|
||
```bash | ||
docker build -t docker.io/my/database:latest . | ||
``` | ||
|
||
Hooray! You have successfully packaged a sanitized MySQL database image! | ||
|
||
### Integrate | ||
|
||
The database image can then be integrated into local development workflows using Docker Compose (or similar). | ||
|
||
Below is an example of how this can be configured. | ||
A toolkit for exporting, sanitizing and packaging MySQL database. | ||
|
||
```bash | ||
$ cat docker-compose.yml | ||
## The Goal of this Project | ||
|
||
--- | ||
version: "3" | ||
To provide developers with the tools to share MySQL database dumps in a safe and repeatable manner. | ||
|
||
services: | ||
## Getting Started | ||
|
||
# Services used as part of the local development environment. | ||
mysql: | ||
image: docker.io/my/database:latest | ||
``` | ||
See our [documentation site](https://mtk.skpr.io) for a tutorial on how to get started. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# Dependencies | ||
/node_modules | ||
|
||
# Production | ||
/build | ||
|
||
# Generated files | ||
.docusaurus | ||
.cache-loader | ||
|
||
# Misc | ||
.DS_Store | ||
.env.local | ||
.env.development.local | ||
.env.test.local | ||
.env.production.local | ||
|
||
npm-debug.log* | ||
yarn-debug.log* | ||
yarn-error.log* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
# Website | ||
|
||
This website is built using [Docusaurus](https://docusaurus.io/), a modern static website generator. | ||
|
||
### Installation | ||
|
||
``` | ||
$ yarn | ||
``` | ||
|
||
### Local Development | ||
|
||
``` | ||
$ yarn start | ||
``` | ||
|
||
This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server. | ||
|
||
### Build | ||
|
||
``` | ||
$ yarn build | ||
``` | ||
|
||
This command generates static content into the `build` directory and can be served using any static contents hosting service. | ||
|
||
### Deployment | ||
|
||
Using SSH: | ||
|
||
``` | ||
$ USE_SSH=true yarn deploy | ||
``` | ||
|
||
Not using SSH: | ||
|
||
``` | ||
$ GIT_USER=<Your GitHub username> yarn deploy | ||
``` | ||
|
||
If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
nickschuch: | ||
name: Nick Schuch | ||
title: Lead Architect | ||
url: https://github.com/nickschuch | ||
image_url: https://github.com/nickschuch.png | ||
page: true | ||
socials: | ||
bluesky: nickschuch | ||
github: nickschuch |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
--- | ||
slug: containerized-databases-for-developers | ||
title: Containerized Databases for Developers | ||
date: 2018-07-07 | ||
authors: [nickschuch] | ||
--- | ||
|
||
The first blog post to introduce the term database images back in 2018, revolutionizing the way databases are packaged and shared with development teams and CI/CD pipelines. | ||
|
||
This blog post highlights how implementing MySQL Toolkit has enhanced and streamlined our development workflow. | ||
|
||
https://www.previousnext.com.au/blog/containerized-databases-developers |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
--- | ||
slug: drupalsouth-2019 | ||
title: Containerized Databases for Developers | ||
date: 2018-12-04 | ||
authors: [nickschuch] | ||
--- | ||
|
||
What would you do if your laptop was stolen? Have I got your attention? Yes? Good :) Now think about all the un-sanitised databases that were on that laptop. | ||
|
||
Lets talk about packaging those up into sanitised, developer consumable artefacts. | ||
|
||
[](https://www.youtube.com/watch?v=W7lE1H-OCRs) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
--- | ||
slug: patterns-local-development-nirvana | ||
title: Patterns for Local Development Nirvana | ||
date: 2024-04-24 | ||
authors: [nickschuch] | ||
--- | ||
|
||
Recorded at DrupalSouth 2024 in Sydney, my session is a deep dive into our journey towards local dev environment nirvana and highlights | ||
how database images levelled up our local development workflows. | ||
|
||
https://www.previousnext.com.au/blog/containerized-databases-developers |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
--- | ||
sidebar_position: 3 | ||
--- | ||
|
||
# AWS RDS | ||
|
||
MySQL Toolkit offers native integration with AWS RDS, enabling developers to seamlessly copy data from an Aurora MySQL-Compatible DB directly into Amazon S3. This can be achieved (under the hood) using the [SELECT INTO OUTFILE S3 statement](https://docs.aws.amazon.com/prescriptive-guidance/latest/archiving-mysql-data/export-data-by-using-select-into-outfile-s3.html). | ||
|
||
## Data flow | ||
|
||
This feature results in a backup and restore workflow that looks like the below. | ||
|
||
### Backup | ||
|
||
```mermaid | ||
graph LR; | ||
MTK --> RDS | ||
RDS -- Store Table Structure --> SQL | ||
RDS -- Export Data --> S3 | ||
``` | ||
|
||
### Restore | ||
|
||
```mermaid | ||
graph LR; | ||
SQL -- Table Structure --> RDS | ||
S3 -- Import Data --> RDS | ||
``` | ||
|
||
## Performance | ||
|
||
By utilizing this approach, we've observed a 30% improvement in performance for database backups and restores on [our hosting platform](https://skpr.com.au). | ||
|
||
## Usage | ||
|
||
Below demonstrates how the `mtk dump` command can be updated to enable AWS RDS integration. | ||
|
||
**Before** | ||
|
||
```bash | ||
mtk dump -u USERNAME -pPASSWORD -h HOSTNAME DATABASE > export.sql | ||
``` | ||
|
||
**After** | ||
|
||
```bash | ||
# Flags | ||
mtk dump --provider=rds --rds-region=ap-southeast-2 --rds-s3-uri=s3://my-s3-bucket -u USERNAME -pPASSWORD -h HOSTNAME DATABASE > export.sql | ||
|
||
# Environment variables | ||
export MTK_PROVIDER=rds | ||
export MTK_RDS_REGION=ap-southeast-2 | ||
export MTK_RDS_S3_URI=s3://my-s3-bucket | ||
mtk dump -u USERNAME -pPASSWORD -h HOSTNAME DATABASE > export.sql | ||
``` |
Oops, something went wrong.