预备条件
-
所有支持的操作系统都使用 64-bit x86 架构。Rancher 兼容当前所有的主流 Linux 发行版。
-
请安装 ntp（Network Time Protocol），以防止在客户端和服务器之间由于时间不同步造成的证书验证错误。
-
某些 Linux 发行版的默认防火墙规则可能会阻止 Kubernetes 集群内的通信。从 Kubernetes v1.19 开始，你必须关闭 firewalld，因为它与 Kubernetes 网络插件冲突。
为什么是三个节点？
在RKE集群中，Rancher服务器数据存储在etcd上。这个etcd数据库在所有三个节点上运行。
etcd数据库需要奇数个节点，这样它总是可以选出一个拥有大多数etcd集群的领导者。如果etcd数据库不能选出一个领导者，etcd可能会遭受分裂的大脑，需要从备份中恢复集群。如果三个etcd节点中的一个失败，剩下的两个节点可以选举一个领导者，因为它们拥有etcd节点总数的大多数。
预备条件
- 安装 kubernetes ，这里我选择 rke2 方式
- 私有镜像仓库：你可以选择安装 harbor 或者 安装 registry
配置私有仓库
(每个rke2节点都要执行更新)
$ vim /etc/rancher/rke2/registries.yaml
mirrors:
docker.io:
endpoint:
- "https://harbor.ghostwritten.com"
configs:
"harbor.ghostwritten.com":
auth:
username: admin
password: Harbor12345
tls:
insecure_skip_verify: true
如果是master 节点，重启 rke2-server
systemctl restart rke2-server.service && systemctl status rke2-server.service
如果是 node 节点，重启 rke2-agent
systemctl restart rke2-agent.service && systemctl status rke2-agent.service
重启后/etc/rancher/rke2/registries.yaml的仓库配置会传递到/var/lib/rancher/rke2/agent/etc/containerd/config.toml
$ cat /var/lib/rancher/rke2/agent/etc/containerd/config.toml |grep -C 4 harbor
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://harbor.ghostwritten.com"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.ghostwritten.com".auth]
username = "admin"
password = "Harbor12345"
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.ghostwritten.com".tls]
insecure_skip_verify = true
介质清单
.
├── cert-manager-images.txt
├── cert-manager-v1.13.3.tgz
├── cert-manager.yaml
├── images
│ ├── quay.io_cert-manager-cainjector_v1.13.3.tar
│ ├── quay.io_cert-manager-controller_v1.13.3.tar
│ ├── quay.io_cert-manager-ctl_v1.13.3.tar
│ └── quay.io_cert-manager-webhook_v1.13.3.tar
├── images.sh
└── rancher-images.txt
安装 helm
wget https://get.helm.sh/helm-v3.13.3-linux-amd64.tar.gz
tar -xzvf helm-v3.13.3-linux-amd64.tar.gz
cp linux-amd64/helm /usr/local/bin/
helm version
rm -rf linux-amd64 helm-v3.13.3-linux-amd64.tar.gz
安装 cert-manager
下载介质
（在联网节点下载）
wget https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.crds.yaml
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm fetch jetstack/cert-manager --version v1.13.3
helm template ./cert-manager-v1.13.3.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> cert-manager-images.txt
cert-manager-images.txt 镜像列表：
quay.io/jetstack/cert-manager-cainjector:v1.13.3
quay.io/jetstack/cert-manager-controller:v1.13.3
quay.io/jetstack/cert-manager-webhook:v1.13.3
quay.io/jetstack/cert-manager-ctl:v1.13.3
镜像入库
修改 images.sh 参数:
- registry_name=‘harbor.ghostwritten.com’
- project=‘cert-manager’
- docker=‘/usr/bin/podman’
- images_list=‘cert-manager-images.txt’
sh images.sh pull
sh images.sh save
#搬运离线节点
sh images.sh load
sh images.sh push
helm 部署
（离线环境）
为 cert-manager 创建命名空间
kubectl create namespace cert-manager
创建crd
$ kubectl apply -f cert-manager.crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
install_cert-manager.sh 内容：
helm install --debug cert-manager ./cert-manager-v1.13.3.tgz \
--namespace cert-manager \
--create-namespace \
--set image.repository=harbor.ghostwritten.com/rancher/cert-manager-controller \
--set webhook.image.repository=harbor.ghostwritten.com/rancher/cert-manager-webhook \
--set cainjector.image.repository=harbor.ghostwritten.com/rancher/cert-manager-cainjector \
--set startupapicheck.image.repository=harbor.ghostwritten.com/rancher/cert-manager-ctl
查看
$ kubectl get pod -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-79bf4c54cf-xplpn 1/1 Running 0 22s
cert-manager-cainjector-6b8d78448f-2j8n4 1/1 Running 0 22s
cert-manager-startupapicheck-grzgz 1/1 Running 0 19s
cert-manager-webhook-c78d5bb7-mkr9x 1/1 Running 0 22s
卸载
$ helm delete cert-manager -n cert-manager
release "cert-manager" uninstalled
$ kubectl get job -n cert-manager
NAME COMPLETIONS DURATION AGE
cert-manager-startupapicheck 1/1 27m 28m
$ kubectl delete job -n cert-manager cert-manager-startupapicheck
job.batch "cert-manager-startupapicheck" deleted
$ kubectl delete ns cert-manager
namespace "cert-manager" deleted
安装 rancher
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
helm fetch rancher-stable/rancher --version=v2.8.0
镜像入库
仅 helm 安装 rancher 依赖的镜像如下 rancher-images.txt ：
docker.io/rancher/fleet-agent:v0.9.0
docker.io/rancher/fleet:v0.9.0
docker.io/rancher/gitjob:v0.1.96
docker.io/rancher/mirrored-cluster-api-controller:v1.4.4
docker.io/rancher/rancher:v2.8.0
docker.io/rancher/rancher-webhook:v0.4.2
docker.io/rancher/shell:v0.1.22
但涉及 rancher 集群管理，比如引导安装多种 rke2、安装插件等依赖的镜像。共470个，参考：
修改 images.sh 参数:
- registry_name=‘harbor.ghostwritten.com’
- project=‘rancher’
- docker=‘/usr/bin/podman’
- images_list=‘rancher-images.txt’
sh images.sh pull
sh images.sh save
#搬运离线节点
sh images.sh load
sh images.sh push
helm 安装
helm install rancher ./rancher-2.8.0.tgz \
--namespace cattle-system \
--create-namespace \
--set hostname=rancher01.ghostwritten.dev \
--set certmanager.version=1.13.3 \
--set rancherImage=harbor.ghostwritten.com/rancher \
--set systemDefaultRegistry=harbor.ghostwritten.com \ # 设置在 Rancher 中使用的默认私有镜像仓库
--set useBundledSystemChart=true # 使用打包的 Rancher System Chart
输出：
# Source: rancher/templates/issuer-rancher.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: rancher
labels:
app: rancher
chart: rancher-2.8.0
heritage: Helm
release: rancher
spec:
ca:
secretName: tls-rancher
NOTES:
Rancher Server has been installed.
NOTE: Rancher may take several minutes to fully initialize. Please standby while Certificates are being issued, Containers are started and the Ingress rule comes up.
Check out our docs at https://rancher.com/docs/
If you provided your own bootstrap password during installation, browse to https://rancher01.ghostwritten.dev to get started.
If this is the first time you installed Rancher, get started by running this command and clicking the URL it generates:
echo https://rancher01.ghostwritten.dev/dashboard/?setup=$(kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}')
To get just the bootstrap password on its own, run:
kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{ "\n" }}'
Happy Containering!
验证
$ helm ls -n cattle-system
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
rancher cattle-system 1 2024-01-10 05:15:14.096529535 -0500 EST deployed rancher-2.8.0 v2.8.0
rancher-webhook cattle-system 1 2024-01-10 10:21:03.85680939 +0000 UTC deployed rancher-webhook-103.0.1+up0.4.2 0.4.2
$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-fleet-system fleet-controller-6b4dd5db6c-shwsp 1/1 Running 0 8m59s
cattle-fleet-system gitjob-75b769c6fb-bx5zg 1/1 Running 0 8m59s
cattle-provisioning-capi-system capi-controller-manager-6c4d64c64-4pjvz 1/1 Running 0 6m15s
cattle-system helm-operation-2jt9g 0/2 Completed 0 7m40s
cattle-system helm-operation-9sgm6 0/2 Completed 0 9m13s
cattle-system helm-operation-pt2w6 0/2 Completed 0 8m9s
cattle-system helm-operation-t2kkr 0/2 Completed 0 7m11s
cattle-system helm-operation-zt929 0/2 Completed 0 6m21s
cattle-system rancher-5ccc6b9d89-hsv6m 1/1 Running 0 9m2s
cattle-system rancher-5ccc6b9d89-ph9l7 1/1 Running 0 12m
cattle-system rancher-5ccc6b9d89-w2h66 1/1 Running 0 12m
cattle-system rancher-webhook-dd69b4d4f-s8n9n 1/1 Running 0 7m
cert-manager cert-manager-79bf4c54cf-xplpn 1/1 Running 0 51m
cert-manager cert-manager-cainjector-6b8d78448f-2j8n4 1/1 Running 0 51m
cert-manager cert-manager-webhook-c78d5bb7-mkr9x 1/1 Running 0 51m
kube-system cloud-controller-manager-rke2-master01 1/1 Running 4 (9m25s ago) 5d2h
kube-system cloud-controller-manager-rke2-master02 1/1 Running 5 (24h ago) 5d1h
kube-system cloud-controller-manager-rke2-master03 1/1 Running 0 120m
kube-system etcd-rke2-master01 1/1 Running 1 5d2h
kube-system etcd-rke2-master02 1/1 Running 1 5d1h
kube-system etcd-rke2-master03 1/1 Running 0 120m
kube-system helm-install-rke2-canal-6v6qr 0/1 Completed 0 5d2h
kube-system helm-install-rke2-coredns-b5ttn 0/1 Completed 0 5d2h
kube-system helm-install-rke2-ingress-nginx-45cqw 0/1 Completed 0 5d2h
kube-system helm-install-rke2-metrics-server-mq6qh 0/1 Completed 0 5d2h
kube-system helm-install-rke2-snapshot-controller-crd-jn4zf 0/1 Completed 0 5d2h
kube-system helm-install-rke2-snapshot-controller-zt8f5 0/1 Completed 2 5d2h
kube-system helm-install-rke2-snapshot-validation-webhook-kgjbt 0/1 Completed 0 5d2h
kube-system kube-apiserver-rke2-master01 1/1 Running 1 5d2h
kube-system kube-apiserver-rke2-master02 1/1 Running 1 5d1h
kube-system kube-apiserver-rke2-master03 1/1 Running 0 120m
kube-system kube-controller-manager-rke2-master01 1/1 Running 5 (9m24s ago) 5d2h
kube-system kube-controller-manager-rke2-master02 1/1 Running 5 (24h ago) 5d1h
kube-system kube-controller-manager-rke2-master03 1/1 Running 0 120m
kube-system kube-proxy-rke2-master01 1/1 Running 1 (24h ago) 5d2h
kube-system kube-proxy-rke2-master02 1/1 Running 1 (24h ago) 5d1h
kube-system kube-proxy-rke2-master03 1/1 Running 0 120m
kube-system kube-proxy-rke2-node01 1/1 Running 0 24h
kube-system kube-scheduler-rke2-master01 1/1 Running 1 (24h ago) 5d2h
kube-system kube-scheduler-rke2-master02 1/1 Running 1 (24h ago) 5d1h
kube-system kube-scheduler-rke2-master03 1/1 Running 0 120m
kube-system rke2-canal-dwr7m 2/2 Running 2 (24h ago) 5d
kube-system rke2-canal-jjbzf 2/2 Running 0 121m
kube-system rke2-canal-kzvc9 2/2 Running 2 (24h ago) 5d1h
kube-system rke2-canal-ssvcb 2/2 Running 2 (24h ago) 5d2h
kube-system rke2-coredns-rke2-coredns-565dfc7d75-6dbr9 1/1 Running 1 (24h ago) 5d2h
kube-system rke2-coredns-rke2-coredns-565dfc7d75-tvf2f 1/1 Running 1 (24h ago) 5d1h
kube-system rke2-coredns-rke2-coredns-autoscaler-6c48c95bf9-lb2xt 1/1 Running 1 (24h ago) 5d2h
kube-system rke2-ingress-nginx-controller-4dhc7 1/1 Running 1 (24h ago) 5d
kube-system rke2-ingress-nginx-controller-8lp6v 1/1 Running 1 (24h ago) 5d2h
kube-system rke2-ingress-nginx-controller-s5rw9 1/1 Running 0 120m
kube-system rke2-ingress-nginx-controller-x2p78 1/1 Running 1 (24h ago) 5d1h
kube-system rke2-metrics-server-c9c78bd66-szclt 1/1 Running 1 (24h ago) 5d2h
kube-system rke2-snapshot-controller-6f7bbb497d-b426h 1/1 Running 1 (24h ago) 5d2h
kube-system rke2-snapshot-validation-webhook-65b5675d5c-2b98t 1/1 Running 1 (24h ago) 5d2h
$ kubectl get ingress -n cattle-system
NAME CLASS HOSTS ADDRESS PORTS AGE
rancher <none> rancher01.ghostwritten.dev 192.168.23.91,192.168.23.92,192.168.23.93,192.168.23.94 80, 443 17m
$ kubectl -n cattle-system rollout status deploy/rancher
deployment "rancher" successfully rolled out
$ kubectl get secret -n cattle-system tls-rancher-ingress -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: ecdsa-with-SHA256
Issuer: O = dynamiclistener-org, CN = dynamiclistener-ca@1704881898
Validity
Not Before: Jan 10 10:18:18 2024 GMT
Not After : Jan 7 10:18:18 2034 GMT
Subject: O = dynamiclistener-org, CN = dynamiclistener-ca@1704881898
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:78:35:e2:95:be:fc:08:70:b0:89:39:77:d6:0e:
5f:5c:30:cc:5c:10:b8:78:55:58:c6:1c:df:58:7b:
8b:75:6c:36:48:08:5a:31:1c:01:be:54:ca:a4:69:
5d:e1:ce:98:a3:05:c5:97:fd:5f:ca:eb:ba:74:21:
bf:e4:ee:10:db
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
CE:E4:D9:15:58:B4:B1:7C:19:34:05:F7:59:52:11:1C:FE:52:4A:79
Signature Algorithm: ecdsa-with-SHA256
30:44:02:20:01:a8:8c:a0:ce:9b:83:1a:17:f3:62:35:e6:80:
94:d6:50:b1:b8:a0:96:44:5e:d0:8b:de:6b:b0:e8:30:ad:d3:
02:20:5d:0a:f0:92:36:4d:41:40:ea:00:7a:b4:de:68:ae:f9:
a7:de:46:eb:90:8c:e7:77:43:4a:d0:af:1a:95:25:58