Skip to content
This repository was archived by the owner on Dec 9, 2022. It is now read-only.

Commit 986460b

Browse files
author
Jeremy Lewi
committed
Forward all issues for Kubeflow org; setup dev environment
* #57 is tracking setting up new staging and prod environments * This PR sets up a new staging (or dev environment) * We create a kustomize manifest for deploying the front end into that namespace * The staging environment is configured to use the dev instance of the issue label bot backend microservice (i.e the pubsub workers) * I created some python scripts to make it easier to setup the secrets. * The motivation for doing this was to test the changes to the front end * Front end now forwards all issues for the kubeflow org to the backend * This is needed because we want to use multiple models for all Kubeflow repos kubeflow/code-intelligence#70 * The backend should also be configured with logging to measure the impact of the predictions. kubeflow/code-intelligence#104 is an a test issue showing that the bot is working. * Fix how keys are handled * For GOOGLE_APPLICATION_CREDENTIALS; depend on that environment variable being set and pointing to the file containing the private key; don't get the private key from an environment variable and then write it to a file. * For the GitHub App private key; use an environment variable to point to the file containing the PEM key. * Create a script to create the secrets. * Flask app is running in dev namespace * create_secrets.py creates secrets needed for dev instance
1 parent 26d8fb6 commit 986460b

18 files changed

+665
-113
lines changed

‎deployment/Dockerfile

+1
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ RUN pip install \
6868
tensorflow==1.12.0 \
6969
seldon-core==0.2.6
7070

71+
7172
COPY requirements.txt .
7273
RUN pip install -r requirements.txt
7374
COPY flask_app flask_app/

‎deployment/README.md

+24-3
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,21 @@ with mlbot.net.
55

66
This is currently running on a GKE cluster.
77

8+
See [machine-learning-apps/Issue-Label-Bot#57](https://github.com/machine-learning-apps/Issue-Label-Bot/issues/57) for a log of how
9+
the service was deployed.
10+
11+
To build a new image
12+
13+
```
14+
skaffold build
15+
```
16+
17+
Then to update the image
18+
19+
```
20+
cd overlays/dev|prod
21+
kustomize edit set image gcr.io/github-probots/label-bot-frontend=gcr.io/github-probots/label-bot-frontend:${TAG}@${SHA}
22+
```
823

924
## github-probots
1025

@@ -40,12 +55,18 @@ Deploying it
4055

4156
There is a staging cluster for testing running in
4257

43-
* **GCP project**: issue-label-bot-dev
44-
* **cluster**: github-mlapp-test
45-
* **namespace**: mlapp
58+
* **GCP project**: github-probots
59+
* **cluster**: kf-ci-ml
60+
* **namespace**: label-bot-dev
4661

4762
Deploying it
4863

64+
1. Create the secrets
65+
66+
67+
68+
TODO(jlewi): instructions below are outdated
69+
4970
1. Create the deployment
5071

5172
```

‎deployment/base/deployment.yaml

+78
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: ml-github-app
5+
labels:
6+
app: ml-github-app
7+
spec:
8+
replicas: 9
9+
selector:
10+
matchLabels:
11+
app: ml-github-app
12+
template:
13+
metadata:
14+
labels:
15+
app: ml-github-app
16+
spec:
17+
containers:
18+
- name: frontend
19+
image: gcr.io/github-probots/label-bot-frontend
20+
command: ["python", "app.py"]
21+
workingDir: "/flask_app"
22+
readinessProbe:
23+
httpGet:
24+
path: /
25+
port: 3000
26+
initialDelaySeconds: 10
27+
periodSeconds: 3
28+
env:
29+
- name: DATABASE_URL
30+
valueFrom:
31+
secretKeyRef:
32+
name: ml-app-inference-secret
33+
key: DATABASE_URL
34+
- name: WEBHOOK_SECRET
35+
valueFrom:
36+
secretKeyRef:
37+
name: ml-app-inference-secret
38+
key: WEBHOOK_SECRET
39+
# The values for the Kubeflow kf-label-bot-dev application
40+
# See kubeflow/code-intelligence#84. This is suitable
41+
# for development but shouldn't be used in production
42+
- name: APP_ID
43+
value: "50112"
44+
# Pato the GitHub app PEM key
45+
- name: GITHUB_APP_PEM_KEY
46+
value: /var/secrets/github/kf-label-bot-dev.private-key.pem
47+
# The GCP project and pubsub topic to publish to.
48+
# Default to the test/dev topic
49+
- name: GCP_PROJECT_ID
50+
value: issue-label-bot-dev
51+
- name: GCP_PUBSUB_TOPIC_NAME
52+
value: TEST_event_queue
53+
- name: GOOGLE_APPLICATION_CREDENTIALS
54+
value: /var/secrets/google/user-gcp-sa.json
55+
- name: FLASK_ENV
56+
value: production
57+
- name: PORT
58+
value: '3000'
59+
- name: APP_URL
60+
value: https://mlbot.net/
61+
- name: authors
62+
value: 'c'
63+
ports:
64+
- containerPort: 443
65+
- containerPort: 80
66+
- containerPort: 3000
67+
volumeMounts:
68+
- name: user-gcp-sa
69+
mountPath: /var/secrets/google
70+
- name: github-app
71+
mountPath: /var/secrets/github
72+
volumes:
73+
- name: user-gcp-sa
74+
secret:
75+
secretName: user-gcp-sa
76+
- name: github-app
77+
secret:
78+
secretName: github-app

‎deployment/base/ingress.yaml

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: extensions/v1beta1
2+
kind: Ingress
3+
metadata:
4+
name: frontend
5+
annotations:
6+
# The ip and certificate name should be overwritten for each
7+
# overlay and set to the correct values
8+
kubernetes.io/ingress.global-static-ip-name: fake-ip
9+
networking.gke.io/managed-certificates: fake-certificate
10+
spec:
11+
backend:
12+
serviceName: ml-github-app
13+
servicePort: 3000
14+
rules:
15+
- http:
16+
paths:
17+
- path: /
18+
backend:
19+
serviceName: ml-github-app
20+
servicePort: 3000

‎deployment/base/kustomization.yaml

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
namePrefix: label-bot-
4+
commonLabels:
5+
app: label-bot
6+
service: label-bot
7+
images:
8+
- name: gcr.io/github-probots/label-bot-frontend
9+
newName: gcr.io/github-probots/label-bot-frontend
10+
resources:
11+
- deployment.yaml
12+
- service.yaml
13+
- ingress.yaml

‎deployment/base/service.yaml

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
name: ml-github-app
5+
labels:
6+
app: ml-github-app
7+
spec:
8+
ports:
9+
- port: 3000
10+
protocol: TCP
11+
selector:
12+
app: ml-github-app
13+
type: NodePort

‎deployment/overlays/dev/certificate.yaml

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: networking.gke.io/v1beta1
2+
kind: ManagedCertificate
3+
metadata:
4+
name: certificate
5+
spec:
6+
domains:
7+
- label-bot-dev.mlbot.net

‎deployment/overlays/dev/deployment.yaml

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: ml-github-app
5+
spec:
6+
replicas: 1
7+
template:
8+
spec:
9+
containers:
10+
- name: frontend
11+
env:
12+
- name: DATABASE_URL
13+
valueFrom:
14+
secretKeyRef:
15+
name: ml-app-inference-secret
16+
key: DATABASE_URL
17+
- name: WEBHOOK_SECRET
18+
valueFrom:
19+
secretKeyRef:
20+
name: ml-app-inference-secret
21+
key: WEBHOOK_SECRET
22+
# The values for the Kubeflow kf-label-bot-dev application
23+
# See kubeflow/code-intelligence#84. This is suitable
24+
# for development but shouldn't be used in production
25+
- name: APP_ID
26+
value: "50112"
27+
# Path the GitHub app PEM key
28+
- name: GITHUB_APP_PEM_KEY
29+
value: /var/secrets/github/kf-label-bot-dev.private-key.pem
30+
# The GCP project and pubsub topic to publish to should
31+
# correspond to the production backend
32+
- name: GCP_PROJECT_ID
33+
value: issue-label-bot-dev
34+
- name: GCP_PUBSUB_TOPIC_NAME
35+
value: TEST_event_queue
36+
- name: GOOGLE_APPLICATION_CREDENTIALS
37+
value: /var/secrets/google/user-gcp-sa.json
38+
- name: FLASK_ENV
39+
value: production
40+

‎deployment/overlays/dev/ingress.yaml

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: extensions/v1beta1
2+
kind: Ingress
3+
metadata:
4+
name: frontend
5+
annotations:
6+
kubernetes.io/ingress.global-static-ip-name: label-bot-dev
7+
networking.gke.io/managed-certificates: certificate

‎deployment/overlays/dev/kustomization.yaml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
bases:
4+
- ../../base
5+
commonLabels:
6+
environment: dev
7+
namespace: label-bot-dev
8+
resources:
9+
- certificate.yaml
10+
patchesStrategicMerge:
11+
- deployment.yaml
12+
- ingress.yaml

‎deployment/overlays/prod/deployment.yaml

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: ml-github-app
5+
spec:
6+
replicas: 9
7+
spec:
8+
containers:
9+
- name: frontend
10+
env:
11+
- name: DATABASE_URL
12+
valueFrom:
13+
secretKeyRef:
14+
name: ml-app-inference-secret
15+
key: DATABASE_URL
16+
- name: WEBHOOK_SECRET
17+
valueFrom:
18+
secretKeyRef:
19+
name: ml-app-inference-secret
20+
key: WEBHOOK_SECRET
21+
# The values for the Kubeflow issue-label-bot application
22+
# that is available in the GitHub marketplace
23+
- name: APP_ID
24+
value: "27079"
25+
# Pato the GitHub app PEM key
26+
- name: GITHUB_APP_PEM_KEY
27+
value: /var/secrets/github/kf-label-bot-dev.private-key.pem
28+
# The GCP project and pubsub topic to publish to should
29+
# correspond to the production backend
30+
- name: GCP_PROJECT_ID
31+
value: issue-label-bot-dev
32+
- name: GCP_PUBSUB_TOPIC_NAME
33+
value: event_queue
34+
- name: GOOGLE_APPLICATION_CREDENTIALS
35+
value: /var/secrets/google/user-gcp-sa.json
36+
- name: FLASK_ENV
37+
value: production
38+

‎developer_guide.md

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Developer guide
2+
3+
1. You can deploy the front end using skaffold
4+
5+
```
6+
skaffold dev --cleanup=False
7+
```
8+
9+
* Your Kubernetes context should be set to using the `github-probots-dev` namespace
10+
* This will continually rebuild and upate your code
11+
* Skaffold's file sync feature is used to update the code in the image without rebuilding and
12+
redeploying
13+
* This makes redeploying very easy.
14+
15+
1. To send a GitHub webhook event you can either open up an issue or you can use `scripts/send_request.py`
16+
17+
* The latter is useful because it avoids needing to open up a new GitHub issue
18+
19+
* Right now the bot is only designed to respond to issues opened events.

0 commit comments

Comments
 (0)