Skip to content

Commit 0ac43b3

Browse files
tinyzimmertinyzimmer
committed
some more examples and docs ; autoload k3p.yaml if found in cwd
1 parent fe014ca commit 0ac43b3

File tree

18 files changed

+394
-28
lines changed

18 files changed

+394
-28
lines changed

‎README.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# k3p
22

3-
A `k3s` packager and installer, primarily intended for airgapped deployments
3+
A `k3s` packager and installer, originally and primarily intended for air-gapped deployments, but could see use cases elsewhere.
44

55
For documentation on `k3p` usage, see the [command docs here](doc/k3p.md).
66

@@ -12,7 +12,9 @@ For documentation on `k3p` usage, see the [command docs here](doc/k3p.md).
1212

1313
## Quickstart
1414

15-
Will publish releases via actions in the future. For now, on a system with `git` and `go` installed.
15+
First download a binary for your system from the [releases](https://github.com/tinyzimmer/k3p/releases) page.
16+
17+
Or, to build from source, on a system with, `make`, `git`, and `go` installed.
1618

1719
```bash
1820
git clone https://github.com/tinyzimmer/k3p
@@ -97,4 +99,5 @@ whoami-5dc4dd9cdf-qvvnz 1/1 Running 0 32s
9799
```
98100
99101
For further information on adding worker nodes and/or setting up HA, you can view the command documentation,
100-
however more complete documentation will come in the future in the form of [examples](examples/).
102+
however more complete documentation will come in the future in the form of [examples](examples/) and other docs.
103+
There are already a few simple examples that you can use to get a general understanding of the workflow.

‎doc/k3p_build.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ k3p build [flags]
1111
```
1212
-a, --arch string The architecture to package the distribution for. Only (amd64, arm, and arm64 are supported) (default "amd64")
1313
-C, --channel string The release channel to retrieve the version of k3s from (default "stable")
14-
-c, --config string An optional config file providing variables to be used at installation
14+
-c, --config string An optional file providing variables and other configurations to be used at installation, if a k3p.yaml in the current directory exists it will be used automatically
1515
-E, --eula string A file containing an End User License Agreement to display to the user upon installing the package
1616
-e, --exclude strings Directories to exclude when reading the manifest directory
1717
--exclude-images Don't include container images with the final archive

‎examples/docker/README.md‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# Playing with Docker
2+
3+
This directory goes into more detail on smoke testing packages with docker.
4+
If you have used [`k3d`](https://github.com/rancher/k3d) in the past most of this will be familiar to you.
5+
6+
To start off, build the package in this directory (for the purpose of these examples we'll exclude images from the archive):
7+
8+
```bash
9+
# Build the package and give it a unique name
10+
$ k3p build --exclude-images --name=k3p-docker
11+
12+
2020/12/14 10:09:59 [INFO] Building package "k3p-docker"
13+
2020/12/14 10:09:59 [INFO] Detecting latest k3s version for channel stable
14+
2020/12/14 10:10:00 [INFO] Latest k3s version is v1.19.4+k3s1
15+
2020/12/14 10:10:00 [INFO] Packaging distribution for version "v1.19.4+k3s1" using "amd64" architecture
16+
2020/12/14 10:10:00 [INFO] Downloading core k3s components
17+
2020/12/14 10:10:00 [INFO] Fetching checksums...
18+
2020/12/14 10:10:00 [INFO] Fetching k3s install script...
19+
2020/12/14 10:10:00 [INFO] Fetching k3s binary...
20+
2020/12/14 10:10:00 [INFO] Skipping bundling k3s airgap images with the package
21+
2020/12/14 10:10:00 [INFO] Validating checksums...
22+
2020/12/14 10:10:00 [INFO] Searching "/home/tinyzimmer/devel/k3p/examples/docker" for kubernetes manifests to include in the archive
23+
2020/12/14 10:10:00 [INFO] Detected kubernetes manifest: "/home/tinyzimmer/devel/k3p/examples/docker/whoami.yaml"
24+
2020/12/14 10:10:00 [INFO] Skipping bundling container images with the package
25+
2020/12/14 10:10:00 [INFO] Writing package metadata
26+
2020/12/14 10:10:00 [INFO] Archiving version "latest" of "k3p-docker" to "/home/tinyzimmer/devel/k3p/examples/docker/package.tar"
27+
```
28+
29+
To install this package to a simple single node cluster running in docker you can do the following:
30+
31+
```bash
32+
# --write-kubeconfig is optional and will extract the kubeconfig once the server is up
33+
# otherwise instructions are printed for fetching it directly from the container
34+
$ k3p install package.tar --docker --write-kubeconfig kubeconfig.yaml
35+
2020/12/14 10:11:13 [INFO] Loading the archive
36+
2020/12/14 10:11:13 [INFO] Creating docker network k3p-docker
37+
2020/12/14 10:11:13 [INFO] Creating docker volume k3p-docker-server-0
38+
2020/12/14 10:11:14 [INFO] Copying the archive to the rancher installation directory
39+
2020/12/14 10:11:14 [INFO] Installing binaries to /usr/local/bin
40+
2020/12/14 10:11:14 [INFO] Installing scripts to /usr/local/bin/k3p-scripts
41+
2020/12/14 10:11:14 [INFO] Installing manifests to /var/lib/rancher/k3s/server/manifests
42+
2020/12/14 10:11:14 [INFO] Running k3s installation script
43+
2020/12/14 10:11:14 [INFO] Starting k3s docker node k3p-docker-server-0
44+
2020/12/14 10:11:15 [INFO] Starting k3s docker node k3p-docker-serverlb
45+
2020/12/14 10:11:15 [INFO] Waiting for server to write the admin kubeconfig
46+
2020/12/14 10:11:17 [INFO] Writing the kubeconfig to "kubeconfig.yaml"
47+
2020/12/14 10:11:17 [INFO] The cluster has been installed
48+
2020/12/14 10:11:17 [INFO] You can view the cluster by running `kubectl --kubeconfig kubeconfig.yaml cluster-info`
49+
50+
$ kubectl --kubeconfig kubeconfig.yaml cluster-info
51+
Kubernetes master is running at https://127.0.0.1:6443
52+
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
53+
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
54+
55+
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
56+
57+
$ kubectl --kubeconfig kubeconfig.yaml get pod
58+
NAME READY STATUS RESTARTS AGE
59+
whoami-5db874f58d-dcx48 1/1 Running 0 54s
60+
61+
# To remove the cluster when you are done
62+
$ k3p uninstall --name=k3p-docker # The --name flag supports tab completion
63+
2020/12/14 10:13:53 [INFO] Removing docker cluster k3p-docker
64+
2020/12/14 10:13:53 [INFO] Removing docker container and volumes for k3p-docker-serverlb
65+
2020/12/14 10:13:53 [INFO] Removing docker container and volumes for k3p-docker-server-0
66+
2020/12/14 10:13:54 [INFO] Removing docker network k3p-docker
67+
```
68+
69+
You can specify server/agent count and configurations also (this is the same as for a regular install)
70+
71+
```bash
72+
$ k3p install package.tar --docker --write-kubeconfig kubeconfig.yaml \
73+
--servers 3 --agents 3 \ # Specify number of server and agent nodes
74+
--k3s-server-arg="--disable=traefik" # can be specified multiple times, there is also an agent equivalent
75+
76+
# ...
77+
# ...
78+
79+
$ kubectl --kubeconfig kubeconfig.yaml get node
80+
NAME STATUS ROLES AGE VERSION
81+
k3p-docker-agent-0 Ready worker 47s v1.19.4+k3s1
82+
k3p-docker-agent-1 Ready worker 50s v1.19.4+k3s1
83+
k3p-docker-agent-2 Ready worker 49s v1.19.4+k3s1
84+
k3p-docker-server-0 Ready etcd,master 55s v1.19.4+k3s1
85+
k3p-docker-server-1 Ready etcd,master 24s v1.19.4+k3s1
86+
k3p-docker-server-2 Ready etcd,master 44s v1.19.4+k3s1
87+
88+
$ kubectl --kubeconfig kubeconfig.yaml get pod -A
89+
NAMESPACE NAME READY STATUS RESTARTS AGE
90+
default whoami-5db874f58d-xtgrl 1/1 Running 0 2m42s
91+
kube-system coredns-66c464876b-sr4fw 1/1 Running 0 2m42s
92+
kube-system local-path-provisioner-7ff9579c6-5wpnq 1/1 Running 0 2m42s
93+
kube-system metrics-server-7b4f8b595-rmwl5 1/1 Running 0 2m42s
94+
```
95+
96+
Forwarding ports to specific nodes in the cluster works the same as `k3d`
97+
98+
```bash
99+
$ k3p install package.tar --docker \
100+
--publish 8080:80@loadbalancer \ # Forward 8080 on the local machine to 80 on the LoadBalancer
101+
--publish 8081:80@server[0] # Forward 8081 on the local machine to 80 on the first server instance
102+
103+
2020/12/14 10:29:10 [INFO] Loading the archive
104+
2020/12/14 10:29:10 [INFO] Creating docker network k3p-docker
105+
2020/12/14 10:29:10 [INFO] Creating docker volume k3p-docker-server-0
106+
2020/12/14 10:29:10 [INFO] Copying the archive to the rancher installation directory
107+
2020/12/14 10:29:10 [INFO] Installing binaries to /usr/local/bin
108+
2020/12/14 10:29:10 [INFO] Installing scripts to /usr/local/bin/k3p-scripts
109+
2020/12/14 10:29:10 [INFO] Installing manifests to /var/lib/rancher/k3s/server/manifests
110+
2020/12/14 10:29:11 [INFO] Running k3s installation script
111+
2020/12/14 10:29:11 [INFO] Starting k3s docker node k3p-docker-server-0
112+
2020/12/14 10:29:11 [INFO] Starting k3s docker node k3p-docker-serverlb
113+
2020/12/14 10:29:12 [INFO] The cluster has been installed
114+
2020/12/14 10:29:12 [INFO] You can retrieve the kubeconfig by running `docker cp k3p-docker-server-0:/etc/rancher/k3s/k3s.yaml ./kubeconfig.yaml`
115+
116+
$ docker ps
117+
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
118+
b90d5ac9107a rancher/k3d-proxy:latest "/bin/sh -c nginx-pr…" 7 seconds ago Up 5 seconds 0.0.0.0:6443->6443/tcp, 0.0.0.0:8080->80/tcp k3p-docker-serverlb
119+
f9d226f0a17b rancher/k3s:v1.19.4-k3s1 "/bin/k3s server --t…" 7 seconds ago Up 6 seconds 0.0.0.0:8081->80/tcp k3p-docker-server-0
120+
```

‎examples/docker/whoami.yaml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
---
2+
apiVersion: apps/v1
3+
kind: Deployment
4+
metadata:
5+
name: whoami
6+
namespace: default
7+
labels:
8+
app: whoami
9+
spec:
10+
replicas: 1
11+
selector:
12+
matchLabels:
13+
app: whoami
14+
template:
15+
metadata:
16+
labels:
17+
app: whoami
18+
annotations:
19+
spec:
20+
containers:
21+
- name: whoami
22+
image: "traefik/whoami:latest"
23+
imagePullPolicy: IfNotPresent
24+
ports:
25+
- name: http
26+
containerPort: 80
27+
---
28+
apiVersion: v1
29+
kind: Service
30+
metadata:
31+
name: whoami
32+
namespace: default
33+
labels:
34+
app: whoami
35+
spec:
36+
type: ClusterIP
37+
ports:
38+
- name: http
39+
port: 80
40+
targetPort: 80
41+
selector:
42+
app: whoami
43+
---
44+
kind: Ingress
45+
apiVersion: extensions/v1beta1
46+
metadata:
47+
name: whoami
48+
namespace: default
49+
spec:
50+
rules:
51+
- host: localhost
52+
http:
53+
paths:
54+
- path: /
55+
backend:
56+
serviceName: whoami
57+
servicePort: 80

‎examples/ha/README.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
## HA Deployments
2+
3+
In terms of the contents of the packag we again use a simple `whoami` example, except with this time specifying pod anti-affinity to ensure pods are not co-located on the same node.
4+
K3p can be used to add new nodes to the cluster either locally via the `install` command, or remotely via the `node add` command.
5+
6+
### Creating the Initial Node
7+
8+
With the experimental k3s embedded etcd HA, one node has to be started with the `--cluster-init` flag, and then additional control-plane instances can be added through joining the initial node.
9+
10+
With the package in this directory already built, SSH in to your first host and run `k3p install` with the `--init-ha` flag. (This can also be done remotely with the `--host` flag).
11+
12+
```bash
13+
[core@coreos1 ~]$ sudo k3p install package.tar --init-ha
14+
15+
2020/12/14 12:52:47 [INFO] Loading the archive
16+
2020/12/14 12:52:48 [INFO] Copying the archive to the rancher installation directory
17+
2020/12/14 12:52:49 [INFO] Generating a node token for additional control-plane instances
18+
2020/12/14 12:52:49 [INFO] Installing binaries to /usr/local/bin
19+
2020/12/14 12:52:49 [INFO] Installing scripts to /usr/local/bin/k3p-scripts
20+
2020/12/14 12:52:49 [INFO] Installing images to /var/lib/rancher/k3s/agent/images
21+
2020/12/14 12:52:50 [INFO] Installing manifests to /var/lib/rancher/k3s/server/manifests
22+
2020/12/14 12:52:50 [INFO] Running k3s installation script
23+
2020/12/14 12:52:50 [K3S] [INFO] Skipping k3s download and verify
24+
2020/12/14 12:52:50 [K3S] [INFO] Skipping installation of SELinux RPM
25+
2020/12/14 12:52:50 [K3S] [INFO] Creating /usr/local/bin/kubectl symlink to k3s
26+
2020/12/14 12:52:50 [K3S] [INFO] Creating /usr/local/bin/crictl symlink to k3s
27+
2020/12/14 12:52:50 [K3S] [INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
28+
2020/12/14 12:52:50 [K3S] [INFO] Creating killall script /usr/local/bin/k3s-killall.sh
29+
2020/12/14 12:52:50 [K3S] [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
30+
2020/12/14 12:52:50 [K3S] [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
31+
2020/12/14 12:52:50 [K3S] [INFO] systemd: Creating service file /etc/systemd/system/k3s.service
32+
2020/12/14 12:52:50 [K3S] [INFO] systemd: Enabling k3s unit
33+
2020/12/14 12:52:50 [K3S] Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
34+
2020/12/14 12:52:51 [K3S] [INFO] systemd: Starting k3s
35+
2020/12/14 12:52:59 [INFO] The cluster has been installed
36+
2020/12/14 12:52:59 [INFO] You can view the cluster by running `k3s kubectl cluster-info`
37+
38+
# A token was generated for joining new control-plane instances during the install.
39+
# A pre-generated one can also be used. To retrieve the generated one you can run
40+
[core@coreos1 ~]$ sudo k3p token get server
41+
fFUiC96GBQ69XgENdvhabseBd53vSUVWuYhrLJKVRX08a3M9RA8qSYypBxLMX0iCEPBnWl6BmZ6WKIw4pAhtbQYhMWveiGI3YbkGMkwJQnTfuTnkBzzMIvsitvBiwqg3
42+
43+
# So far we have a single node, and we are unable to schedule two of our pods
44+
[core@coreos1 ~]$ sudo k3s kubectl get node
45+
NAME STATUS ROLES AGE VERSION
46+
coreos1 Ready etcd,master 68s v1.19.4+k3s1
47+
48+
[core@coreos1 ~]$ sudo k3s kubectl get pod
49+
NAME READY STATUS RESTARTS AGE
50+
whoami-5f47859667-87l9q 1/1 Running 0 63s
51+
whoami-5f47859667-l77k6 0/1 Pending 0 63s
52+
whoami-5f47859667-n8jvh 0/1 Pending 0 63s
53+
```
54+
55+
To join a second and third instance to the cluster there are two (actually three) ways we can do this. The first way is to install the package again to the other instances, using the `--join` flag to signal joining an existing cluster.
56+
57+
```bash
58+
[core@coreos2 ~]$ sudo k3p install package.tar \
59+
--join https://172.18.64.84:6443 \ # The IP and API port of the first instance
60+
--join-role server \ # Join as a server instance (the default option is as an agent and uses a different token)
61+
--join-token fFUiC96GBQ69XgENdvhabseBd53vSUVWuYhrLJKVRX08a3M9RA8qSYypBxLMX0iCEPBnWl6BmZ6WKIw4pAhtbQYhMWveiGI3YbkGMkwJQnTfuTnkBzzMIvsitvBiwqg3
62+
63+
# ...
64+
# ...
65+
```
66+
67+
You can also use `k3p node add` from the initial node to bring in new instances using SSH. If you have public key authentication setup you can use that, otherwise it will prompt for a password.
68+
69+
You can also do this from a remote instance with the `--leader` flag assuming it uses the same SSH credentials as the new node you are adding.
70+
71+
```bash
72+
[core@coreos1 ~]$ sudo k3p node add 172.18.64.91 \ # The remote address of the node
73+
--ssh-user core \ # The user to use for SSH
74+
--private-key ~/.ssh/id_rsa \ # The SSH private key (or omit to be prompted for a password)
75+
--node-role server # Join as a server
76+
77+
# ...
78+
# ...
79+
```
80+
81+
Once that is done you will have a highly available cluster and deployment
82+
83+
```bash
84+
[core@coreos1 ~]$ sudo k3s kubectl get node
85+
NAME STATUS ROLES AGE VERSION
86+
coreos1 Ready etcd,master 11m v1.19.4+k3s1
87+
coreos2 Ready etcd,master 6m20s v1.19.4+k3s1
88+
coreos3 Ready etcd,master 2m57s v1.19.4+k3s1
89+
90+
[core@coreos1 ~]$ sudo k3s kubectl get pod
91+
NAME READY STATUS RESTARTS AGE
92+
whoami-5f47859667-87l9q 1/1 Running 0 11m
93+
whoami-5f47859667-l77k6 1/1 Running 0 11m
94+
whoami-5f47859667-n8jvh 1/1 Running 0 11m
95+
```

‎examples/ha/whoami.yaml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
apiVersion: apps/v1
3+
kind: Deployment
4+
metadata:
5+
name: whoami
6+
namespace: default
7+
labels:
8+
app: whoami
9+
spec:
10+
replicas: 3
11+
selector:
12+
matchLabels:
13+
app: whoami
14+
template:
15+
metadata:
16+
labels:
17+
app: whoami
18+
annotations:
19+
spec:
20+
containers:
21+
- name: whoami
22+
image: "traefik/whoami:latest"
23+
imagePullPolicy: Never
24+
ports:
25+
- name: http
26+
containerPort: 80
27+
affinity:
28+
podAntiAffinity:
29+
requiredDuringSchedulingIgnoredDuringExecution:
30+
- labelSelector:
31+
matchExpressions:
32+
- key: app
33+
operator: In
34+
values:
35+
- whoami
36+
topologyKey: "kubernetes.io/hostname"
37+
---
38+
apiVersion: v1
39+
kind: Service
40+
metadata:
41+
name: whoami
42+
namespace: default
43+
labels:
44+
app: whoami
45+
spec:
46+
type: ClusterIP
47+
ports:
48+
- name: http
49+
port: 80
50+
targetPort: 80
51+
selector:
52+
app: whoami
53+
---
54+
kind: Ingress
55+
apiVersion: extensions/v1beta1
56+
metadata:
57+
name: whoami
58+
namespace: default
59+
spec:
60+
rules:
61+
- host: localhost
62+
http:
63+
paths:
64+
- path: /
65+
backend:
66+
serviceName: whoami
67+
servicePort: 80

examples/kvdi/README.md renamed to examples/helm-charts/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Then build a package in this directory using the provided config:
1818

1919
```sh
2020
# For the sake of producing a smaller artifact, we'll use the --exclude-images flag
21-
$ k3p build -c config.yaml --exclude-images --name kvdi
21+
$ k3p build --exclude-images --name kvdi
2222
2020/12/11 20:32:08 [INFO] Building package "kvdi"
2323
2020/12/11 20:32:08 [INFO] Detecting latest k3s version for channel stable
2424
2020/12/11 20:32:09 [INFO] Latest k3s version is v1.19.4+k3s1
@@ -35,4 +35,4 @@ $ k3p build -c config.yaml --exclude-images --name kvdi
3535
2020/12/11 20:32:09 [INFO] Skipping bundling container images with the package
3636
2020/12/11 20:32:09 [INFO] Writing package metadata
3737
2020/12/11 20:32:09 [INFO] Archiving version "latest" of "kvdi" to "/home/tinyzimmer/devel/k3p/examples/kvdi/package.tar"
38-
```
38+
```

examples/kvdi/config.yaml renamed to examples/helm-charts/k3p.yaml

File renamed without changes.

0 commit comments

Comments
 (0)