Skip to content

Commit

Permalink
modified the documentation with the new Makefile commands
Browse files Browse the repository at this point in the history
  • Loading branch information
frouioui committed Dec 18, 2020
1 parent 421f350 commit b5cb83e
Show file tree
Hide file tree
Showing 12 changed files with 401 additions and 394 deletions.
6 changes: 2 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -127,16 +127,14 @@ setup_jaeger: $(shell chmod +x ./scripts/jaeger.sh)
./scripts/jaeger.sh
kubectl create -n observability -f ./kubernetes/jaeger/operator.yaml
kubectl create -n observability -f ./kubernetes/jaeger/jaeger.yaml
kubectl create -n observability -f ./kubernetes/jaeger/jaeger_ui_ingress_route.yaml
kubectl apply -f kubernetes/init_cluster_vitess_sharded_final_jaeger.yaml
kubectl apply -f kubernetes/traefik/traefik_deployment_jaeger.yaml

setup_traefik:
kubectl create -f kubernetes/traefik/traefik_crd.yaml
kubectl create -f kubernetes/traefik/traefik_rbac.yaml
@echo Wait 5s
@sleep 5
kubectl create -f kubernetes/traefik/traefik_deployment.yaml
kubectl create -f kubernetes/traefik/traefik_deployment_jaeger.yaml
kubectl create -n observability -f ./kubernetes/jaeger/jaeger_ui_ingress_route.yaml

setup_traefik_vitess: $(shell chmod +x ./kubernetes/traefik/vitess/build.sh)
./kubernetes/traefik/vitess/build.sh
Expand Down
94 changes: 19 additions & 75 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,88 +18,32 @@ This project is part of the **Distributed Database Systems** class of the Advanc
- Efficient execution of the data insert, update, and queries
- Monitoring of the whole distributed system

----
## Quick Start

In this quick start we will cover the following items:

- Setup our Kubernetes cluster
- Create a sharded Vitess cluster
- Add Traefik Proxy to our cluster
- Enable tracing with Jaeger
- Setup some APIs
- Setup a Frontend application
- Setup some monitoring with Grafana, Prometheus and Alertmanager

Once the quick start is over, we will have a fully setup application using distributed database systems.

### Requirements
Before we start, there are some requirements:
## Requirements
To run locally, tagenal needs:

- Have at least 10Gb of available RAM on the host
- Have Kubernetes / Minikube installed
- Have Golang version 1.15.x installed
- Install [yq](https://github.com/mikefarah/yq) a YAML processor
- Install [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler) (jb), allowing us to deal with jsonnet files
- Install jsonnet using your system's packet manager
- Install vtctlclient, the following command `go get vitess.io/vitess/go/cmd/vtctlclient` can be used
- Install `jsonnet` using your system's packet manager
- Install `vtctlclient`, the following command `go get vitess.io/vitess/go/cmd/vtctlclient` can be used
- Install `mysql` and `mysql-client` using your system's packet manager
- Have Kubernetes / Minikube installed
- Have Golang version 1.15.x installed
- Install gojsontoyaml, the following command `go get github.com/brancz/gojsontoyaml` can be used
- Run the shell scripts that are located in `./lib/*.sh`. These scripts will download the necessary basic libraries and repositories

### Steps
The quick start is as follows:
1. [Setup of a simple cluster](./docs/setup-minikube-vitess.md)
2. [Setup of our sharded Vitess cluster](./docs/setup-sharded-vitess-cluster.md)
3. [Setup monitoring](./docs/setup-monitoring.md)
4. [Setup Jaeger](./docs/setup-jaeger.md)
5. [Start the APIs](./docs/setup-apis.md)
6. [Start the frontend](./docs/setup-frontend.md)
- Install `gojsontoyaml`, the following command `go get github.com/brancz/gojsontoyaml` can be used
- Run the shell scripts that are located in `./lib/*.sh`. These scripts will download the necessary basic libraries and repositories

---
## Quick Start

## Data models
In this quick start we will cover the following items:

Tagenal is composed of 2 main tables for now. They are:
1. [Setup the Kubernetes cluster](./docs/setup-minikube-vitess.md)
1. [Setup Jaeger](./docs/setup-jaeger.md)
1. [Setup Traefik Proxy]()
1. [Setup the Vitess cluster]()
1. [Setup the Redis cluster]()
1. [Setup monitoring with Grafana, Prometheus and Alertmanager](./docs/setup-monitoring.md)
1. [Setup the APIs](./docs/setup-apis.md)
1. [Setup the Frontend application](./docs/setup-frontend.md)

### user table
```sql
CREATE TABLE user (
_id INT NOT NULL auto_increment,
timestamp CHAR(14) DEFAULT NULL,
id CHAR(5) DEFAULT NULL,
uid CHAR(5) DEFAULT NULL,
name CHAR(9) DEFAULT NULL,
gender CHAR(7) DEFAULT NULL,
email CHAR(10) DEFAULT NULL,
phone CHAR(10) DEFAULT NULL,
dept CHAR(9) DEFAULT NULL,
grade CHAR(7) DEFAULT NULL,
language CHAR(3) DEFAULT NULL,
region VARBINARY(256),
role CHAR(6) DEFAULT NULL,
preferTags CHAR(7) DEFAULT NULL,
obtainedCredits CHAR(3) DEFAULT NULL,
PRIMARY KEY(_id)
);
```
Once the quick start is over, we will have a fully setup application using distributed database systems.

### article table
```sql
CREATE TABLE article (
_id INT NOT NULL auto_increment,
timestamp CHAR(14) DEFAULT NULL,
id CHAR(7) DEFAULT NULL,
aid CHAR(7) DEFAULT NULL,
title CHAR(15) DEFAULT NULL,
category VARBINARY(256) DEFAULT NULL,
abstract CHAR(30) DEFAULT NULL,
articleTags CHAR(14) DEFAULT NULL,
authors CHAR(40) DEFAULT NULL,
language CHAR(3) DEFAULT NULL,
text TEXT(500) DEFAULT NULL,
image CHAR(255) DEFAULT NULL,
video CHAR(255) DEFAULT NULL,
PRIMARY KEY(_id)
);
```
95 changes: 21 additions & 74 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Tagenal

[![Build Status](https://travis-ci.com/frouioui/tagenal.svg?token=XhmJBhJBxshbY6hsWepE&branch=master)](https://travis-ci.com/frouioui/tagenal)

## Description of the project
<img align="right" width="100" height="100" src="./Tsinghua_University_Logo.png">

Expand All @@ -16,86 +18,31 @@ This project is part of the **Distributed Database Systems** class of the Advanc
- Efficient execution of the data insert, update, and queries
- Monitoring of the whole distributed system

----
## Quick Start

In this quick start we will cover the following items:

- Setup our Kubernetes cluster
- Create a sharded Vitess cluster
- Add Traefik Proxy to our cluster
- Enable tracing with Jaeger
- Setup some APIs
- Setup a Frontend application
- Setup some monitoring with Grafana, Prometheus and Alertmanager

Once the quick start is over, we will have a fully setup application using distributed database systems.

### Requirements
Before we start, there are some requirements:
## Requirements
To run locally, tagenal needs:

- Have at least 10Gb of available RAM on the host
- Have Kubernetes / Minikube installed
- Have Golang version 1.15.x installed
- Install [yq](https://github.com/mikefarah/yq) a YAML processor
- Install [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler) (jb), allowing us to deal with jsonnet files
- Install vtctlclient, the following command `go get vitess.io/vitess/go/cmd/vtctlclient` can be used
- Install `jsonnet` using your system's packet manager
- Install `vtctlclient`, the following command `go get vitess.io/vitess/go/cmd/vtctlclient` can be used
- Install `mysql` and `mysql-client` using your system's packet manager
- Have Kubernetes / Minikube installed
- Have Golang version 1.15.x installed
- Run the shell scripts that are located in `./lib/*.sh`. These scripts will download the basic libraries and repositories that we need

### Steps
The quick start is architectured as followed:
1. [Setup of a simple cluster](./setup-minikube-vitess.md)
2. [Setup our sharded Vitess cluster](./setup-sharded-vitess-cluster.md)
3. [Setup monitoring](./setup-monitoring.md)
4. [Setup Jaeger](./setup-jaeger.md)
5. [Start the APIs](./setup-apis.md)
6. [Start the frontend](./setup-frontend.md)
- Install `gojsontoyaml`, the following command `go get github.com/brancz/gojsontoyaml` can be used
- Run the shell scripts that are located in `./lib/*.sh`. These scripts will download the necessary basic libraries and repositories

---

## Data models
## Quick Start

Tagenal is composed of 2 main tables for now. They are:
In this quick start we will cover the following items:

### user table
```sql
CREATE TABLE user (
_id INT NOT NULL auto_increment,
timestamp CHAR(14) DEFAULT NULL,
id CHAR(5) DEFAULT NULL,
uid CHAR(5) DEFAULT NULL,
name CHAR(9) DEFAULT NULL,
gender CHAR(7) DEFAULT NULL,
email CHAR(10) DEFAULT NULL,
phone CHAR(10) DEFAULT NULL,
dept CHAR(9) DEFAULT NULL,
grade CHAR(7) DEFAULT NULL,
language CHAR(3) DEFAULT NULL,
region VARBINARY(256),
role CHAR(6) DEFAULT NULL,
preferTags CHAR(7) DEFAULT NULL,
obtainedCredits CHAR(3) DEFAULT NULL,
PRIMARY KEY(_id)
);
```
1. [Setup the Kubernetes cluster](./docs/setup-minikube-vitess.md)
1. [Setup Jaeger](./docs/setup-jaeger.md)
1. [Setup Traefik Proxy]()
1. [Setup the Vitess cluster]()
1. [Setup the Redis cluster]()
1. [Setup monitoring with Grafana, Prometheus and Alertmanager](./docs/setup-monitoring.md)
1. [Setup the APIs](./docs/setup-apis.md)
1. [Setup the Frontend application](./docs/setup-frontend.md)

### article table
```sql
CREATE TABLE article (
_id INT NOT NULL auto_increment,
timestamp CHAR(14) DEFAULT NULL,
id CHAR(7) DEFAULT NULL,
aid CHAR(7) DEFAULT NULL,
title CHAR(15) DEFAULT NULL,
category VARBINARY(256) DEFAULT NULL,
abstract CHAR(30) DEFAULT NULL,
articleTags CHAR(14) DEFAULT NULL,
authors CHAR(40) DEFAULT NULL,
language CHAR(3) DEFAULT NULL,
text TEXT(500) DEFAULT NULL,
image CHAR(255) DEFAULT NULL,
video CHAR(255) DEFAULT NULL,
PRIMARY KEY(_id)
);
```
Once the quick start is over, we will have a fully setup application using distributed database systems.
73 changes: 73 additions & 0 deletions docs/setup-api-frontend.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Quick Start - 6. Setup the APIs and frontend

In this final section, we are going to setup two APIs and a frontend application. We will deploy them to the `default` namespace of our kubernetes cluster.

We will cover how to build, push, and run the frontend application and APIs. The frontend code can be found [here](./api/users/README.md), and the APIs can be found here:

- `users` [[see](./api/users/)]
- `articles` [[see](./api/articles/)]

## Frontend

### Build and push

After modifying the frontend code, and creating a new version. We can build and push the new version onto a docker registry. We can modify the name of the docker image and the repository in the frontend's Makefile.

```
make build_push_frontend
```

### Run the frontend on kubernetes

To run the frontend on our kubernetes cluster, we use the following command:

```
make run_frontend_k8s
```

> To use another image than the default one, we need to change the kubernetes manifests in: `./kubernetes/frontend/frontend.yaml`, and specify the new image name/repository.
We can access the frontend at this URL: http://tagenal


### Stop the frontend

To stop the frontend, use the following command:

```
make stop_frontend_k8s
```


## APIs

### Build and Push docker images

After modifying the codebase. A new version of the docker image can be built and pushed to a public docker repository. We do so using the following command:

```
make build_push_apis
```

### Run the APIs on kubernetes

To run the APIs on our kubernetes cluster we use the following command. This will create the two APIs in the `default` namespace of our kubernetes cluster.

```
make run_apis_k8s
```

> To use another image than the default ones, we can change the kubernetes manifests in: `./kubernetes/api/**/*_api_server.yaml`, and specify the proper image names.
Now that our APIs are up and running, we can access them using the ingress routes that were automatically defined in the above command. The URLs are:

- http://api.tagenal/users
- http://api.tagenal/articles

### Stop the APIs

To stop the APIs we run the following command:

```
make stop_apis_k8s
```
32 changes: 0 additions & 32 deletions docs/setup-frontend.md

This file was deleted.

14 changes: 8 additions & 6 deletions docs/setup-jaeger.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,21 @@
# Setup Jaeger
# Quick Start - 2. Setup Jaeger

This section will cover how to setup the `observability` namespace of our kubernetes cluster.
This section will cover how to setup our first kubernetes namespace: `observability`. This namespace will host our observability stack, for now only composed of Jaeger.

## Setup Jaeger

We use the following command for a few things. It will create the whole `observability` namespace and the required CRDs. It will also add Jaeger configuration to the Vitess cluster and to Traefik.
The following command will do a few things for us. It will create the `observability` namespace, and the CRDs that Jaeger needs to successfully run on kubernetes. Then, it will

```
make setup_jaeger
```

Tagenal is mostly experiment-oriented and not production-oriented. We then use the `AllInOne` configuration of Jaeger, which will greatly ease the deployment.

The command also created the ingress route to access Jaeger dashboard. Which can be access at this URL: http://jaeger.tagenal. On this interface we can see the trace of our services.
By using minikube dashboard or `kubectl` we can observe the successful deployment of Jaeger.

## Next step
## Next

The next step is to run the APIs in our kubernetes cluster. Which is detailed [in the next section](./setup-apis.md).
For now, we can't access Jaeger UI, we will cover that in the next section thanks to Traefik Proxy. Next step is [here](./setup-traefik.md).

<!-- The next step is to run the APIs in our kubernetes cluster. Which is detailed [in the next section](./setup-apis.md). -->
Loading

0 comments on commit b5cb83e

Please sign in to comment.