1.集群介绍

1.1 服务版本信息

服务名 版本信息
kubelet 1.24.2
kubeadm 1.24.2
kubectl 1.24.2

1.2 为什么弃用docker?

DockerKubernetes 使用的第一个容器运行时。 这也是许多 Kubernetes 用户和爱好者如此熟悉Docker的原因之一。 对 Docker 的支持被硬编码到 Kubernetes 中——一个被项目称为 dockershim 的组件。

随着容器化成为行业标准,Kubernetes 项目增加了对其他运行时的支持。 最终实现了容器运行时接口(CRI),让系统组件(如 kubelet)以标准化的方式与容器运行时通信。 因此,dockershim 成为了 Kubernetes 项目中的一个异常现象。

Dockerdockershim 的依赖已经渗透到 CNCF 生态系统中的各种工具和项目中,这导致了代码脆弱。

通过删除 dockershim CRI,我们拥抱了 CNCF 的第一个价值: “快比慢好”。

原文查看: https://kubernetes.io/zh-cn/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/

all节点: 代表所有节点所在服务器都需要安装。

2.添加k8s软件源(all节点)

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

3. 安装 kubeadm、kubelet、kubectl(all节点)

$ yum install kubelet kubeadm kubectl -y

4. 启动kubelet服务(all节点)

# 设置开机启动
$ systemctl enable kubelet.service --now

5. 初始化集群(master节点

5.1 配置文件

a. 下载默认配置文件

# 创建目录 /usr/local/kubernetes
$ mkdir -p /usr/local/kubernetes
# 下载默认配置
$ kubeadm config print init-defaults --component-configs KubeletConfiguration > kubeadm.yaml

b. 修改配置文件

# 修改1: advertiseAddress
localAPIEndpoint:
advertiseAddress: 192.168.148.130 # 指定master节点内网IP
...
# 修改2:修改master节点名称
nodeRegistration:
...
name: master # 修改master节点名称


# 修改3:新增 kubeproxy.config节点,并把kube-proxy模式为ipvs,默认为iptables
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
...
# 修改4: 设置imageRepository为阿里云的registry,避免因gcr被墙,无法直接拉取镜像
imageRepository: registry.aliyuncs.com/google_containers
...

# 修改5: 指定k8s版本号,默认这里忽略了小版本号
kubernetesVersion: 1.24.2
...

# 修改6: 确认设置kubelet的cgroupDriver为systemd
---
apiVersion: kubelet.config.k8s.io/v1beta1
...
cgroupDriver: systemd

# 修改7: 指定 pod 子网
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
podSubnet: 10.244.0.0/16 # 指定 pod 子网

5.2 拉取容器镜像

# 根据配置文件拉取镜像
$ kubeadm config images pull --config /usr/local/kubernetes/kubeadm.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.7
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.3-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6

5.3 启动集群

# 根据配置文件启动镜像
$ kubeadm init --config /usr/local/kubernetes/kubeadm.yaml
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[WARNING Hostname]: hostname "node" could not be reached
[WARNING Hostname]: hostname "node": lookup node on 202.106.0.20:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node] and IPs [10.96.0.1 192.168.148.130]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost node] and IPs [192.168.148.130 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost node] and IPs [192.168.148.130 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 10.508558 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node node as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.148.130:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:c82ee852e2e5e276c5c60b836cb1fa154e630ecb0a98dd0b73984b82cbbfa420

6. 添加节点

6.1 加入集群(在节点执行)

# 在 `node1`和`node2`执行以下命令,加入集群
$ kubeadm join 192.168.148.130:6443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:bc891a4b5977f8d86c6e49aeffdecdb95db48183615831042c2462019d03419b
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

6.2 节点报错localhost:8080 was refused

# 报错信息
$ kubectl get nodes
W0720 18:18:30.467593 6025 loader.go:221] Config not found: /etc/kubernetes/admin.conf
The connection to the server localhost:8080 was refused - did you specify the right host or port?

原因是kubectl命令需要使用kubernetes-admin来运行;

解决方法如下:将主节点中的/etc/kubernetes/admin.conf文件拷贝到从节点相同目录下,然后配置环境变量

a. 解决方法:

# 1.拷贝主节点/etc/kubernetes/admin.conf文件到从节点/etc/kubernetes/admin.conf

# 2.配置环境变量
$ echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
# 3.立即生效
$ source ~/.bash_profile

b. 再次查看

[root@node2 ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane 9m48s v1.24.2
node1 NotReady <none> 8m28s v1.24.2
node2 NotReady <none> 8m2s v1.24.2

NotReady是因为没有安装网络组件。

7.包管理器helm

7.1 什么是helm

每个成功的软件平台都有一个优秀的打包系统,比如Debian、Ubuntu的apt,Red Hat、CentOS的yum。Helm则是Kubernetes上的包管理器。

Helm到底解决了什么问题?为什么Kubernetes需要Helm

答案是:Kubernetes能够很好地组织和编排容器,但它缺少一个更高层次的应用打包工具,而Helm就是来干这件事的。

7.2 安装

# 下载包
[root@master tmp]$ wget https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz
# 解压
[root@master tmp]$ tar -zxvf helm-v3.9.0-linux-amd64.tar.gz
# 移动二进制未加
[root@master tmp]$ mv linux-amd64/helm /usr/local/bin/
# 验证
[root@master tmp]# helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.17.5"}

8. 使用helm部署Calico

选择calico作为k8sPod网络组件,下面使用helmk8s集群中安装calico

8.1 下载helm chart

[root@master helm] $ wget https://github.com/projectcalico/calico/releases/download/v3.23.1/tigera-operator-v3.23.1.tgz

8.2 安装

# 指定安装在命名空间为kube-system下
[root@master helm] $ helm install calico tigera-operator-v3.23.1.tgz -n kube-system
#
$ kubectl get pod -n kube-system | grep tigera-operator

# 所有都变成runing
[root@master helm]$ kubectl get pods -n calico-system
calico-kube-controllers-68884f975d-wt2c2 1/1 Running 0 119s
calico-node-68k58 1/1 Running 0 119s
calico-node-n6dgb 1/1 Running 0 119s
calico-node-p56rl 1/1 Running 0 119s
calico-typha-57b74d64dc-p7ztw 1/1 Running 0 119s
calico-typha-57b74d64dc-rpvzb 1/1 Running 0 110s

8.3 查看当时节点状态(全部变成Ready)

[root@master helm]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 20m v1.24.2
node1 Ready <none> 16m v1.24.2
node2 Ready <none> 15m v1.24.2

9.验证DNS是否可用

# 运行并进入验证容器
$ kubectl run curl --image=radial/busyboxplus:curl -it

# 验证
[ root@curl:/ ]$ nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name: kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

10.使用helm部署ingress-nginx

为了便于将集群中的服务暴露到集群外部,需要使用Ingress。接下来使用Helmingress-nginx部署到Kubernetes上。 Nginx Ingress Controller被部署在Kubernetes的边缘节点上。

10.1 设置边缘节点

这里将node1(192.168.148.131)作为边缘节点,打上Label

[root@master kubernetes]$ kubectl label node node1 node-role.kubernetes.io/edge=
node/node1 labeled

10.2 下载ingress-nginxhelm chart

$ wget https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-4.1.2/ingress-nginx-4.1.2.tgz

10.3 编写install-ingress.yaml

controller:
ingressClassResource:
name: nginx
enabled: true
default: true
controllerValue: "k8s.io/ingress-nginx"
admissionWebhooks:
enabled: false
replicaCount: 1
image:
registry: docker.io
image: unreachableg/k8s.gcr.io_ingress-nginx_controller
tag: "v1.2.0"
digest: sha256:314435f9465a7b2973e3aa4f2edad7465cc7bcdc8304be5d146d70e4da136e51
hostNetwork: true
nodeSelector:
node-role.kubernetes.io/edge: ''
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nginx-ingress
- key: component
operator: In
values:
- controller
topologyKey: kubernetes.io/hostname
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: PreferNoSchedule

nginx ingress controller的副本数replicaCount为1,将被调度到node1这个边缘节点上。这里并没有指定nginx ingress controller service的externalIPs,而是通过hostNetwork: true设置nginx ingress controller使用宿主机网络。

10.4 安装

[root@master kubernetes]$ helm install ingress-nginx ingress-nginx-4.1.2.tgz --create-namespace -n ingress-nginx -f install-ingress.yaml
NAME: ingress-nginx
LAST DEPLOYED: Sun Jul 24 00:01:37 2022
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'

...

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls

测试访问http://192.168.148.131返回默认的nginx 404页,则部署完成。

11. 使用helm部署DashBoard

为了提供更丰富的用户体验,Kubernetes还开发了一个基于WebDashboard,用户可以用Kubernetes Dashboard部署容器化的应用、监控应用的状态、执行故障排查任务以及管理Kubernetes的各种资源。

Kubernetes Dashboard中可以查看集群中应用的运行状态,也能够创建和修改各种Kubernetes资源,比如Deployment、Job、DaemonSet等。用户可以Scale Up/Down Deployment、执行Rolling Update、重启某个Pod或者通过向导部署新的应用。Dashboard能显示集群中各种资源的状态以及日志信息。可以说,**Kubernetes Dashboard提供了kubectl的绝大部分功能**。

11.1 部署metrics-server

Metrics-Server是集群核心监控数据的聚合器。通俗地说,它存储了集群中各节点的监控数据,并且提供了API以供分析和使用。

a. 下载配置文件

# 下载配置文件
[root@master ~]$ wget https://github.com/kubernetes-sigs/metrics-server/releases/download/metrics-server-helm-chart-3.8.2/components.yaml

b. 修改配置

vim components.yaml

...
---
apiVersion: apps/v1
kind: Deployment
metadata:
...
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
#修改1:添加容器启动参数,方便跳过安全tls,生产不推荐使用
- --kubelet-insecure-tls
#修改2:更换镜像地址
image: docker.io/unreachableg/k8s.gcr.io_metrics-server_metrics-server:v0.6.1
imagePullPolicy: IfNotPresent
...

c. 部署

# 部署资源
[root@master kubernetes]$ kubectl apply -f components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
# 查看状态
[root@master kubernetes]$ kubectl get pod -n kube-system | grep metrics
metrics-server-77cffb4988-gmbjd 1/1 Running 0 3m29s

11.2 添加对应的chart repo

# 添加chart
[root@master kubernetes]$ helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
"kubernetes-dashboard" has been added to your repositories

# 更新包
[root@master kubernetes]$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kubernetes-dashboard" chart repository
Update Complete. ⎈Happy Helming!⎈

11.3 定制char配置

image:
repository: kubernetesui/dashboard
tag: v2.5.1
ingress:
enabled: true
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
hosts:
- hui.k8s.com # 自定义域名
tls:
- secretName: example-com-tls-secret
hosts:
- hui.k8s.com
metricsScraper:
enabled: true

11.4  安装部署

$ helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard \
-n kube-system \
-f dashboard.yaml

11.5 访问

a. 编辑hosts文件

# 编写本地 /etc/hosts 将hui.k8s.com指向一个node
192.168.148.131 hui.k8s.com

b. 访问

c. 创建管理token

# 创建管理员  kube-dashboard-admin-hui 
$ kubectl create serviceaccount kube-dashboard-admin-hui -n kube-system

# 在整个集群中为管理员 kube-dashboard-admin-hui 授权
$ kubectl create clusterrolebinding kube-dashboard-admin-hui \
--clusterrole=cluster-admin --serviceaccount=kube-system:kube-dashboard-admin-hui

# 创建集群管理员登录dashboard所需token:
$ kubectl create token kube-dashboard-admin-hui -n kube-system --duration=87600h
eyJhbGciOiJSUzI1NiIsImtpZCI6IkNNTWJmWHNkWXFERWhnbWtJWnB5TzNwYTB5OTJPM3cyUG52VjlRM3NGWGMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxOTc0MDgwNjg0LCJpYXQiOjE2NTg3MjA2ODQsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJrdWJlLWRhc2hib2FyZC1hZG1pbi1odWkiLCJ1aWQiOiJkMjNlMTg2NC1lZWY5LTQ0Y2MtOWIwNC1mZDU1ODk5MjU3ZmMifX0sIm5iZiI6MTY1ODcyMDY4NCwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmt1YmUtZGFzaGJvYXJkLWFkbWluLWh1aSJ9.gcztARblypyGUau_gUtwpdInNWwqL6qwKpCEpgiDkRJUCav4v3zEOPHnXhBlKRXi55HiA5gPsiSZdht5HpzGX8iWqXtdY3s2H72gNQQplbJoDAXXEwA1urBOAZ8X2A-B2yAFFndPvjmzjhd9ag3qGm3GZfFrKs9RcaZM-VeWjF7gQy6j0OR4cvjG913wn9POqeJr_zzSE6WNgqwD0VjYb7O9hln2VtCOi_o2Si-L3VyeVql-epnFaQIwNBJjlHyxulLeSZR-CWFbPv3vFlqDCrIgf81YSY0CS8-R6Axw3IAYxBCZi1Id-vrNQZc84IaLDBAlPfcMQ9cm2mBCi6Nklw

再次访问