Kubernetes cluster_with_Rancher
- Kubernete cluster with rancher
A kubernetes cluster consists of one or more nodes. A node is a host system, whether physical or virtual, with a container runtime and its dependencies (i.e. docker mostly) and several kubernetes system components, that is connected to a network that allows it to reach other nodes in the cluster.
A Kubernetes cluster has two key components:
-
Master (or Control) Nodes A Kubernetes Master is where you direct API calls to services that control the activities of the pods, replications controllers, services, nodes and other components of a Kubernetes cluster. Typically, those calls are made by running kubectl commands. From the Master, containers are deployed to run on Nodes. The Master Nodes expose an API through which the command-line tools and rich clients submit a job, which contains the definition of an application. Each application consists of one or more containers, the storage definitions and the internal and external ports through which they are exposed
-
Worker Nodes. The Worker Nodes are the workhorses of a Kubernetes cluster that are responsible for running the containerized applications.
The control plane running on Master Nodes schedules the containers in one of the Worker Nodes. When an application is scaled, the control plane launches additional containers on any of the available Worker Nodes.
Pods are defined in configuration files (in YAML or JSON formats).Once the containers, system and services are in place, you use the kubectl command to deploy those containers so they run on the Kubernetes Node.
Kubernetes has six main components that form a functioning cluster:
- API server
- Scheduler
- Controller manager
- kubelet
- kube-proxy
- etcd
Each of these components can run as standard Linux processes, or they can run as Docker containers.
- kubelet – the primary node agent that watches for pods that have been assigned to its node and performs actions to keep it healthy and functioning e.g. mount pod volumes, download pod secrets, run containers, perform health checks etc
- kube-proxy – enables the Kubernetes service abstraction by maintaining network rules on the host and performing connection forwardin
Setting up reliable and highly available Kubernetes clusters will require a number of steps
Make the master node services reliable
Set up a redundant storage layer for etcd
Use a highly available load balancer for the Kubernetes API services
Setup multiple master nodes and configure a master election strategy
All our hosts, or cluster nodes are Linux VM with Fedora Atomic installed.
High availability TCP/layer 4 need at least this machines:
- 3 etcd nodes
- 2 conrol nodes
- 1 worker node. This one is called master in some documents.
They all can ssh each others on port 42660.
The default Docker package from Atomic is listed as
- be sure each host is perfectly clean. To achieve this, run the following script:
/root/bin/cleandocker.sh
----------------------------------
docker rm -f $(docker ps -qa)
docker volume rm $(docker volume ls -q)
cleanupdirs=(/var/lib/etcd /etc/kubernetes /etc/cni /opt/cni /var/lib/cni /var/run/calico /var/etcd/backups)
for dir in ${cleanupdirs[@]}; do
echo "Removing $dir"
rm -rf $dir
done
docker system prune -a
mkdir /etc/cni /opt/cni
chcon -Rt svirt_sandbox_file_t /etc/cni
chcon -Rt svirt_sandbox_file_t /opt/cni- needed by SELinux
# mkdir /opt/cni /etc/cni
# chcon -Rt svirt_sandbox_file_t /etc/cni
# chcon -Rt svirt_sandbox_file_t /opt/cni
# semanage port -a -t http_port_t -p tcp 2380
# semanage port -a -t http_port_t -p tcp 2379
# semanage port -a -t http_port_t -p tcp 6443
# semanage fcontext -m -t container_var_run_t /var/run/docker.sock
# restorecon -vF /var/run/docker.sockDisable firewalld
The complete list of needed ports is shown here
If you already have Linux machines deployed and just want to add them into Rancher, click the Custom icon in adding hosts.
The first time a host os added, one has to set up the Host Registration URL. This setup determines what DNS name or IP address, and port the hosts will be connected to the Rancher API.
With each host, there is the ability to add labels to help organize the hosts. The labels are added as an environment variable when launching the rancher/agent container. The host label in the UI will be a key/value pair and the keys must be unique identifiers. When a host is added to Rancher, a rancher agent container is launched on the host. Rancher will automatically pull the correct image version tag for the rancher/agent and run the required version.
For any hosts that are added, ensure that any security groups or firewalls allow traffic. If these are not enabled, the Rancher functionality will be limited.
Use kubeconfig files to organize information about clusters, users, namespaces, and authentication mechanisms. The kubectl command-line tool uses kubeconfig files to find the information it needs to choose a cluster and communicate with the API server of a cluster. By default,kubectl looks for a file named config in the $HOME/.kube directory.
- get information about the cluster:
# kubectl cluster-info
Kubernetes master is running at https://83.XXX.YYY.ZZZ/k8s/clusters/cluster-bx5jz
KubeDNS is running at https://83.XXX.YYY.ZZZ/k8s/clusters/cluster-bx5jz/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.-
List components
-
full debug:
# kubectl cluster-info dump > mylog.txt- list all the nodes of the running Kubernetes cluster:
# kubectl get pods -o wide --all-namespaces
MESPACE NAME READY STATUS RESTARTS AGE IP NODE
cattle-system cattle-cluster-agent-69bdb7cb4b-tpxf8 1/1 Running 0 32m 10.42.1.4 control1
cattle-system cattle-node-agent-d9swf 1/1 Running 0 32m 10.52.11.199 control2
cattle-system cattle-node-agent-hr9rz 1/1 Running 0 32m 10.52.11.193 control1
cattle-system cattle-node-agent-w5kqk 1/1 Running 0 31m 10.52.16.22 worker
ingress-nginx default-http-backend-58fb448f9f-5hth5 1/1 Running 0 38m 10.42.1.2 control1
ingress-nginx nginx-ingress-controller-4zjc6 1/1 Running 0 31m 10.52.16.22 worker
ingress-nginx nginx-ingress-controller-bmsts 1/1 Running 0 38m 10.52.11.199 control2
ingress-nginx nginx-ingress-controller-rjq9g 1/1 Running 0 38m 10.52.11.193 control1
kube-system canal-27ltn 3/3 Running 0 32m 10.52.11.177 etcd1
kube-system canal-8dtwz 3/3 Running 0 32m 10.52.11.191 etcd3
kube-system canal-kdngh 3/3 Running 0 38m 10.52.11.199 control2
kube-system canal-r4zgh 3/3 Running 0 38m 10.52.11.179 etcd2
kube-system canal-rmfld 3/3 Running 0 31m 10.52.16.22 worker
kube-system canal-z4k58 3/3 Running 0 38m 10.52.11.193 control1
kube-system kube-dns-6f7666d48c-pjc9w 3/3 Running 0 32m 10.42.1.3 control1
kube-system kube-dns-6f7666d48c-x8mwc 3/3 Running 0 38m 10.42.0.2 control2
kube-system kube-dns-autoscaler-6bbfff8c44-m2zdp 1/1 Running 0 38m 10.42.0.3 control2Kubernetes allocates an IP address to each pod. When creating a cluster, you need to allocate a block of IPs for Kubernetes to use as Pod IPs. The simplest approach is to allocate a different block of IPs to each node in the cluster as the node is added. A process in one pod should be able to communicate with another pod using the IP of the second pod.
internet Static Public IP | NAT on port 443 [ 10.52.] | Listening on port 443 [ Ingress ]
Container Network Interface (CNI), is a library definition and a set of tools. kubenet, Kubernetes default networking provider, is a very basic network provider but does not have very many feature. Rancher propose 3 network providers:
- Calico
- Canal (Flannel + Calico)
- flannel
Reading this blog, Canal is a solution for anyone who wants to get up and running while taking advantage of familiar technologies that they may already be using.
By default, Docker uses host-private networking. It creates a virtual bridge, called docker0 by default, and allocates a subnet from one of the private address blocks defined in RFC1918 for that bridge. For each container that Docker creates, it allocates a virtual Ethernet device (called veth) which is attached to the bridge.
$ ip a
3: docker0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1460 qdisc noqueue state UP group default
link/ether 02:42:26:2f:c6:9d brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
........
152: veth147a41e@if151: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue master docker0 state UP group default
link/ether 32:d9:72:32:c9:93 brd ff:ff:ff:ff:ff:ff link-netnsid 0
......
169: vethrdb4ea2c055@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 22:6b:bc:93:0a:43 brd ff:ff:ff:ff:ff:ff link-netnsid 5Flannel is a very simple overlay network that satisfies the Kubernetes requirements.
kube-dns which is responsible for DNS service discovery. If the kube-dns add on has been set up properly you can access K8s services using their names directly. You don’t need to remember VIP:PORT combination. The name of the service will suffice. How is this possible? Well, when you use kube-dns, K8s “injects” certain nameservice lookup configuration into new pods that allows you to query the DNS records in the cluster.
From inside a container, run the following command:
root@mywebserver-c5c7d7c96-48g9g/# cat /etc/resolv.conf
nameserver 10.43.0.10
search web-http.svc.cluster.local svc.cluster.local cluster.local infomaniak.ch
options ndots:5The Docker cp command allow you to copy files.
To copy files from container to host, run :
docker cp <containerId>:/file/path/within/container /host/path/targetlonghorn, GlusterFS
{
"Type": "volume",
"Name": "8290b76014a553951caa76d5206880ec171d1e91727a5220cdc44b4648780ff8",
"Source": "/storage/docker_images/volumes/8290b76014a553951caa76d5206880ec171d1e91727a5220cdc44b4648780ff8/_data",
"Destination": "/opt/rke",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"HostConfig": {
"Binds": [
"/etc/kubernetes:/etc/kubernetes:z",
"/etc/cni:/etc/cni:ro,z",
"/opt/cni:/opt/cni:ro,z",
"/var/lib/cni:/var/lib/cni:z",
"/etc/resolv.conf:/etc/resolv.conf",
"/sys:/sys:rprivate",
"/var/lib/docker:/var/lib/docker:rw,rprivate,z",
"/var/lib/kubelet:/var/lib/kubelet:shared,z",
"/var/run:/var/run:rw,rprivate",
"/run:/run:rprivate",
"/etc/ceph:/etc/ceph",
"/dev:/host/dev:rprivate",
"/var/log/containers:/var/log/containers:z",
"/var/log/pods:/var/log/pods:z"
--volume-plugin-dir=/var/lib/kubelet/volumepluginsWhat is an Ingress? It’s a load balancer: it receives incoming HTTP and HTTPS traffic respectively on port 80 and 443 and redirects it to the appropriate service inside the Kubernetes cluster.
Kubernetes ingress is a collection of routing rules that govern how external users access services running in a Kubernetes cluster. Ingress is an API resource which represents a set of traffic routing rules that map external traffic to K8s services. Ingress allows external traffic to land in the cluster in a particular service.
When the SSL certificate was wrongly configured (i’ve put CA Authority chain in certificate), the ingress controller returned a blank web page containing:default backend 404
There are four general strategies in Kubernetes for ingress.
You can expose a Service in these ways:
1- Kubernete Proxy
2- Using a Kubernetes service of type NodePort, which exposes the application on a port across each of your nodes
3- Use a Kubernetes Ingress Resource, or Ingress controller
4- Use a Kubernetes service of type LoadBalancer, which creates an external load balancer that points to a Kubernetes service in your cluster
Reverse proxy servers and load balancers are components in a client-server computing architecture. Both act as intermediaries in the communication between the clients and servers, performing functions that improve efficiency.
The basic definitions are simple:
- A reverse proxy accepts a request from a client, forwards it to a server that can fulfill it, and returns the server’s response to the client.
- A load balancer distributes incoming client requests among a group of servers, in each case returning the response from the selected server to the appropriate client.
NOTE:
- Loadbalancer Layer-4/Layer-7: Layer-4 load balancer allows you to forward both HTTP and TCP traffic, when Layer-7 only forwards HTTP and HTTPS traffic.
- Layer-4 with Rancher is not supported with bare metal servers (our case, VM). RKE clusters deploys the Nginx Ingress Controller.
Load balancers are most commonly deployed when a site needs multiple servers because the volume of requests is too much for a single server to handle efficiently. Deploying multiple servers also eliminates a single point of failure, making the website more reliable. Most commonly, the servers all host the same content, and the load balancer’s job is to distribute the workload in a way that makes the best use of each server’s capacity, prevents overload on any server, and results in the fastest possible response to the client.
We can think of the reverse proxy as a website’s “public face.” Its address is the one advertised for the website, and it sits at the edge of the site’s network to accept requests from web browsers and mobile apps for the content hosted at the website.
benefits are:
- Increased security
- Increased scalability and flexibility
- web acceleration
Load balancing is a method to distribute workloads across multiple computing resources.
Load balancing across multiple application instances is a commonly used technique for optimizing resource utilization, maximizing throughput, reducing latency, and ensuring fault-tolerant configurations. It is possible to use nginx as a very efficient HTTP load balancer to distribute traffic to several application servers and to improve performance, scalability and reliability of web applications with nginx.
We will thus be using Nginx as our Layer 4 Load Balancer (TCP) Nginx will forward all connections to one of your Rancher nodes. In this configuration, the load balancer is positioned in front of your Linux hosts. The load balancer can be any host that you have available that’s capable of running Nginx.
**!! WARNING !!: do not use one of our Rancher nodes as the load balancer. **
We will install Nginx on Dahlia. The machine is a fedora server.
Nginx Ingress Controller exposes the external IP of all nodes that run the Nginx Ingress Controller. You can do either of the following:
1 - Configure your own DNS to map (via A records) your domain name to the IP addresses exposes by the Layer-7 load balancer.
2 - Ask Rancher to generate an xip.io host name for your ingress rule. Rancher will take one of your exposed IPs, say a.b.c.d, and generate a host name ..a.b.c.d.xip.io.
The benefit of using* xip.io* is that you obtain a working entrypoint URL immediately after you create the ingress rule. Setting up your own domain name, on the other hand, requires you to configure DNS servers and wait for DNS to propagate.
4 - define passive Health Checks
When NGINX considers a server unavailable, it temporarily stops sending requests to the server until it is considered active again. The following parameters to the server directive configure the conditions under which NGINX considers a server unavailable:
- max_fails – Sets the number of consecutive failed attempts after which NGINX marks the server as unavailable.
- fail_timeout – Sets the time during which the number of failed attempts specified by the max_fails parameter must happen for the server to be considered unavailable, and also the length of time that NGINX considers the server unavailable after it is marked so.
Below configuration file is for a simple Nginx load balancer with least-connected and HTTPS enabled. Least-connected allows controlling the load on application instances more fairly in a situation when some of the requests take longer to complete.
Test configuration file: # nginx -c /etc/nginx/nginx.conf -t
Reload configuration : # nginx -s reload
Proxying is typically used to distribute the load among several servers, seamlessly show content from different websites, or pass requests for processing to application servers over protocols other than HTTP.
We will have at least two clusters: rancher and TTH. To manage these cluster from our workstation with the kubectl command, we need configuration files, sometimes called kubeconfig file.
A configuration file describes clusters, users, and contexts.
The kube_config_rancher-cluster.yml created after the creation of the rancher cluster is a cluster configuration file. The content is rather similar to the kubeconfig file downloaded from Rancher UI when visiting the cluster dashboard.
These configuration files are placed in /developement/rancher/kubeconfig directory. The config-global is for both Rancher and TTH. As we manage more than one cluster, we need to give some option to kubectl command to tell which cluster.
# kubectl --kubeconfig config-local get nodes
NAME STATUS ROLES AGE VERSION
83.166.144.84 Ready controlplane,etcd,worker 3d v1.10.1
83.166.150.230 Ready controlplane,etcd,worker 3d v1.10.1
83.166.154.37 Ready controlplane,etcd,worker 3d v1.10.1
# kubectl --kubeconfig config-tth get nodes
NAME STATUS ROLES AGE VERSION
control1 Ready controlplane 3d v1.10.5
etcd1 Ready etcd 3d v1.10.5
worker Ready worker 3d v1.10.5The $KUBECONFIG environment variable is a list of paths to configuration files. By default, on our workstation, ~/.kube directory is the place for these files, but they are in /developement/rancher/kubeconfig directory. We edited /etc/profile and exported $KUBECONFIG variable.
For convenience, we have created two root aliases defined in /root/.config/zsh/zshrc.d/02_alias:
alias kube-rancher='kubectl --kubeconfig /developement/rancher/kubeconfig/config-local'
alias kube-tth='kubectl --kubeconfig /developement/rancher/kubeconfig/config-tth'This page contains general commands to run and test the cluster.
# kubectl --kubeconfig etc/kube_config_rancher-cluster.yml get nodes
NAME STATUS ROLES AGE VERSION
83.166.144.84 Ready etcd 51m v1.10.1
83.166.150.230 Ready worker 51m v1.10.1
83.166.154.37 Ready controlplane 51m v1.10.1
# kubectl --kubeconfig etc/kube_config_rancher-cluster.yml get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-system cattle-859b6cdc6b-l9twr 1/1 Running 0 51m
ingress-nginx default-http-backend-564b9b6c5b-pdlsn 1/1 Running 0 51m
ingress-nginx nginx-ingress-controller-8d4rj 1/1 Running 0 51m
ingress-nginx nginx-ingress-controller-v8j7l 1/1 Running 0 51m
kube-system canal-6qsfx 3/3 Running 0 51m
kube-system canal-8qtqp 3/3 Running 0 51m
kube-system canal-rgfsf 3/3 Running 0 51m
kube-system kube-dns-5ccb66df65-qc9gm 2/3 Running 0 51m
kube-system kube-dns-autoscaler-6c4b786f5-q2s8h 1/1 Running 0 51m- Connection is refused on a host
# curl -k -H "Host: rancher.thetradinghall.com" https://83.166.144.85
curl: (7) Failed to connect to 83.166.144.85 port 443: Connection refusedSee if something is listening on this port:
Run on the host netstat -tnlp | grep :443. If no output, nothing is listening.
Verify if a node answer:
curl -k -H "Host: rancher.thetradinghall.com" https://83.166.144.84kubectl is the main command. Find all options in this cheatsheet
- Inspect the deployement:
# kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nginx 1 1 1 1 1hInformations given by the above command:
NAME lists the names of the Deployments in the cluster.
DESIRED displays the desired number of replicas of the application, which you define when you create the Deployment. This is the desired state.
CURRENT displays how many replicas are currently running.
UP-TO-DATE displays the number of replicas that have been updated to achieve the desired state.
AVAILABLE displays how many replicas of the application are available to your users.
AGE displays the amount of time that the application has been running.
- Complete view of the cluster:
# kubectl get pods -o wide --all-namespaces- See Nginx listening:
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 14h
mywebserver NodePort 10.43.151.84 <none> 80:30877/TCP 13- Check No services [and thus no pods] are running in the default K8s space except for the kubernetes API service:
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 4d
# kubectl get pods
No resources found.NOTE: adding -n to kubectl allow to specify the namespace. Thus, to get logs, one can run this:
# kubectl -n <NAMSPACE> logs <NAME>
- Get more information in json format
# kubectl get -o json pod nginx-68688cc77f-rx7kb- List all resources with different types
# kubectl get all
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/nginx 1 1 0 1 30m
NAME DESIRED CURRENT READY AGE
rs/nginx-68688cc77f 1 1 1 30m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/nginx 1 1 0 1 30m
NAME DESIRED CURRENT READY AGE
rs/nginx-68688cc77f 1 1 1 30m
NAME READY STATUS RESTARTS AGE
po/nginx-68688cc77f-rx7kb 1/1 Running 0 30m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 1d
svc/nginx NodePort 10.43.101.138 <none> 80:32527/TCP 30m- general information about kubelet. The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod.
# docker inspect kubelet kubectl --kubeconfig etc/kube_config_rancher-cluster.yml get nodes
NAME STATUS ROLES AGE VERSION
83.166.144.85 Ready etcd 1h v1.10.1
83.166.150.230 Ready worker 1h v1.10.1
83.166.154.37 Ready controlplane 1h v1.10.1The k8s-kubectl-pod-network-test script will help you debug network issues between pods.
- first list your pods by namespace:
# kubectl -n <namespace> get pod .....- log into the pod:
kubectl -n <namespace> exec -it <pod name> -- /bin/bashTIP : If a Pod has more than one Container, use --container or -c to specify a Container in the kubectl exec command.
You may find more completed information on deployements on kubernetes website and in the Kubectl cheatsheet
- Red Hat container orchestration with Kubernetes
- Container technology wiki
- Nginx Ingress controller
- HAProxy
- Creating a Custom Cluster from Scratch
- Introduction to kubernetes
- Building High-Availability Clusters
- Deploy Nginx on a K8s cluster
- Kubernetes clusters for the hobbyist
- Most common errors
- Digital ocean closer look at kubernetes
- Digital ocean getting start with kubernetes
- running 3 etcd nodes
- Run HA cluster on Core OS
- Install rancher load balancer
- ssl certificate authority
- cert manager
https://codefresh.io/kubernetes-tutorial/intro-minikube-kubernetes-entities/
Kube-Master, Kube-worker
https://platform9.com/docs/deploy-kubernetes-the-ultimate-guide/
https://techbeacon.com/one-year-using-kubernetes-production-lessons-learned
Data store
When running a data store such as MongoDB or MySQL, you most likely want the data to be persistent. Out of the box, containers lose their data when they restart. This is fine for stateless components, but not for a persistent data store. Kubernetes has the concept of volumes to work with persistent data.