OpenShift — 部署 OKD 4.5

目录

文档资料

  • 官方文档:https://docs.okd.io/

  • Github:https://github.com/openshift/okd

  • OKD 镜像地址:

    • https://quay.io/repository/openshift/okd
    • https://quay.io/repository/openshift/okd-content
  • openshift-client、openshift-install 下载地址:https://github.com/openshift/okd/releases

  • 官方裸金属部署文档:https://docs.okd.io/latest/installing/installing_bare_metal/installing-bare-metal.html

裸机拓扑

在这里插入图片描述

服务器环境

机器系统地址/hostname服务
Bastion NodeCentOS7.7192.168.120.101 / bastion.okd.example.comCoreDNS、HAProxy、Nginx、Container Image Registry
Bootstrap NodeFedora coreos192.168.120.102 / bootstrap.okd.example.combootstrap
Master NodeFedora coreos192.168.120.103 / master.okd.example.comopenshift-master
Worker NodeFedora coreos192.168.120.104 / worker.okd.example.comopenshift-worker

前期准备

  • 时间同步:各节点的 BIOS 硬件时间(hwclock)必须一致。
  • U 盘启动:各节点的 BIOS 配置成可以通过 U 盘安装操作系统。
  • 节点间 SSH 免密登录
# 创建 SSH 密钥
$ ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/id_rsa

# 启动 ssh-agent 进程为后台任务
$ eval "$(ssh-agent -s)"

# 将 SSH 私钥添加到 ssh-agent
$ ssh-add ~/.ssh/id_rsa

NOTE:OpenShift 部署支持通过 “裸金属接管” 的方式进行部署,但由于需要使用到 TFTP/FTP + DHCP + IPMI 等方式,考虑到复杂度和对现有环境的侵入性,所以本次部署使用 U 盘启动的方式完成。

部署 Bastion Node(堡垒机节点)

Bastion Node 作为我们的 “部署堡垒机”,采用常见的 CentOS 操作系统即可,我们会在 Bastion Node 上安装部署 OpenShift 集群时所需要的 oc CLI 等软件工具支撑,建议连接办公网,作为 Remote SSH 的入口。

基础配置

# 主机名配置
$ hostnamectl set-hostname bastion.okd.example.com

# Selinux 配置
$ sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
$ setenforce 0

# 关闭防火墙
$ systemctl disable firewalld
$ systemctl stop firewalld

安装 OpenShift CLI

OpenShift CLI,简称 oc(openshift-client),下载地址:https://github.com/openshift/okd/releases,本次使用 OKD 4.5 版本,最新 GA tag 为 4.5.0-0.okd-2020-07-14-153706-ga。

$ wget https://github.com/openshift/okd/releases/download/4.5.0-0.okd-2020-07-14-153706-ga/openshift-client-linux-4.5.0-0.okd-2020-07-14-153706-ga.tar.gz

$ tar -zxvf openshift-client-linux-4.5.0-0.okd-2020-07-14-153706-ga.tar.gz

$ cp oc /usr/local/bin/

# 检查版本
$ oc version 

安装 openshift-install

$ wget https://github.com/openshift/okd/releases/download/4.5.0-0.okd-2020-07-14-153706-ga/openshift-install-linux-4.5.0-0.okd-2020-07-14-153706-ga.tar.gz

$ tar -zxvf openshift-install-linux-4.5.0-0.okd-2020-07-14-153706-ga.tar.gz

$ cp openshift-install /usr/local/bin/

# 检查版本
$ openshift-install version

安装 ETCD

$ yum install -y etcd

$ systemctl enable etcd --now

安装 CoreDNS

OpenShift 集群部署需要使用到 Domain Name 来完成 Node Discovery。


$ wget https://github.com/coredns/coredns/releases/download/v1.6.9/coredns_1.6.9_linux_amd64.tgz

$ tar zxvf coredns_1.6.9_linux_amd64.tgz

$ mv coredns /usr/local/bin

$ useradd coredns -s /sbin/nologin
  • 创建 systemd 配置:
$ vi /etc/systemd/system/coredns.service

[Unit]
Description=CoreDNS DNS server
Documentation=https://coredns.io
After=network.target
 
[Service]
PermissionsStartOnly=true
LimitNOFILE=1048576
LimitNPROC=512
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
User=coredns
WorkingDirectory=~
ExecStart=/usr/local/bin/coredns -conf=/etc/coredns/Corefile
ExecReload=/bin/kill -SIGUSR1 
Restart=on-failure
 
[Install]
WantedBy=multi-user.target
  • 编辑 CoreDNS 配置:
$ vi /etc/coredns/Corefile

.:53 {                                              # 监听 53 端口
    template IN A apps.okd.example.com {
        match .*apps\.okd\.example\.com               # 匹配 DNS Query Request Domain Name 的正则表达式
        answer "{{ .Name }} 60 IN A 172.18.22.243"  # 配置 Domain Name 应答的 IP 地址
        fallthrough
    }
    etcd {                              # 配置启用 etcd 插件,后面可以指定域名,例如:etcd test.com
        path /skydns                    # 默认路径为 /skydns,后续所有的 DNS 记录都存储在该路径下
        endpoint http://localhost:2379  # etcd 访问地址,使用空格分隔多个 endpoint
        fallthrough
    }
    prometheus                          # 监控插件
    cache 160
    loadbalance                         # 负载均衡,开启 DNS 轮训查询测试
    forward . 114.114.114.114
    log                                 # 打印日志
}


$ systemctl enable coredns --now

# 验证
$ dig +short apps.okd.example.com @127.0.0.1
  • 添加 OpenShift Cluster Node 的 DNS 解析记录,每条记录都是部署过程中多需要的:
$ cat /etc/resolv.conf
# Generated by NetworkManager
search okd.example.com
nameserver 192.168.120.101

$ alias etcdctlv3='ETCDCTL_API=3 etcdctl'

# Bastion API Server HA
$ etcdctlv3 put /skydns/com/example/okd/api '{"host":"192.168.120.101", "ttl":60}'
$ etcdctlv3 put /skydns/com/example/okd/api-int '{"host":"192.168.120.101", "ttl":60}'

# Bastion Container Images Registry
$ etcdctlv3 put /skydns/com/example/okd/registry '{"host":"192.168.120.101", "ttl":60}'

# Master ETCD
$ etcdctlv3 put /skydns/com/example/okd/etcd-0 '{"host":"192.168.120.103", "ttl":60}'

$ etcdctlv3 put /skydns/com/example/okd/_tcp/_etcd-server-ssl/x1 '{"host":"etcd-0.okd.example.com", "ttl":60, "priority":0, "weight":10, "port":2380}'

$ etcdctlv3 put /skydns/com/example/okd/bastion '{"host":"192.168.120.101", "ttl":60}'
$ etcdctlv3 put /skydns/com/example/okd/bootstrap '{"host":"192.168.120.102", "ttl":60}'
$ etcdctlv3 put /skydns/com/example/okd/master '{"host":"192.168.120.103", "ttl":60}'
$ etcdctlv3 put /skydns/com/example/okd/worker '{"host":"192.168.120.104", "ttl":60}'

安装 HAProxy

$ yum install haproxy -y

$ vi /etc/haproxy/haproxy.cfg
...
listen stats
    bind :9000
    mode http
    stats enable
    stats uri /
    monitor-uri /healthz

frontend openshift-api-server                    # OpenShift API Server (HA)
    bind *:6443
    default_backend openshift-api-server
    mode tcp
    option tcplog

backend openshift-api-server
    balance source
    mode tcp
    server bootstrap 192.168.120.102:6443 check  # API Server in Bootstrap Node
    server master 192.168.120.103:6443 check     # API Server in Master Node

frontend machine-config-server                   # OpenShift Machine Config Server (HA)
    bind *:22623
    default_backend machine-config-server
    mode tcp
    option tcplog

backend machine-config-server
    balance source
    mode tcp
    server bootstrap 192.168.120.102:22623 check # Machine Config Server in Bootstrap Node
    server master 192.168.120.103:22623 check    # Machine Config Server in Master Node
    
$ systemctl enable haproxy && systemctl restart haproxy

安装 Registry

  • 自签发证书,确保客户端安全接入镜像仓库:
# 自建 CA 中心
$ mkdir -p /opt/registry/{auth,certs,data}
$ cd /opt/registry/certs

# 自签发证书,域名为 registry.okd.example.com
$ openssl req -subj '/CN=registry.okd.example.com/O=My Company Name LTD./C=US' -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout domain.key -out domain.crt

# 将自签名的证书复制到默认信任证书路径
$ cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/
$ update-ca-trust extract
  • 生成镜像仓库密钥:
# 为镜像仓库生成密钥
$ echo -n 'admin:admin' | base64 -w0    # YWRtaW46YWRtaW4=

$ vi pull-secret.json
{
	"auths": {
		"registry.okd.example.com:5000": {  # 本地仓库访问地址
			"auth": "YWRtaW46YWRtaW4=",     # 密钥信息
			"email": ""
		}
	}
}

# 检查 Registry 是否正常工作
$ curl -u admin:admin -k https://registry.okd.example.com:5000/v2/_catalog

NOTE:OCP 需要从 Redhat 官网下载密钥,地址:https://cloud.redhat.com/openshift/install/pull-secret;而 OKD 则可以自己生成。

  • 下载官方镜像:
$ yum -y install podman httpd httpd-tools vim

# 设置变量
export OKD_RELEASE="4.5.0-0.okd-2020-07-14-153706-ga"
export LOCAL_REGISTRY='registry.okd.example.com:5000'
export LOCAL_REPOSITORY='openshift/okd'
export PRODUCT_REPO='openshift'
export LOCAL_SECRET_JSON='/root/pull-secret.json'
export RELEASE_NAME="okd"

# 拉取镜像
$ oc adm -a ${LOCAL_SECRET_JSON} release mirror \
     --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OKD_RELEASE} \
     --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
     --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OKD_RELEASE}

# 加载证书、密钥,启动 regsitry 服务。
$ podman run --name example-registry -p 5000:5000 \
     -v /opt/registry/data:/var/lib/registry:z \
     -v /opt/registry/auth:/auth:z \
     -v /opt/registry/certs:/certs:z \
     -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
     -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
     -d docker.io/library/registry:2

安装 Nginx

OpenShift 集群部署采用类似 “boot from Network” 的方式,需要从 NFS(网络文件服务)服务器上下载 CoreOS Image 和 Ignition 文件。

$ yum install -y nginx

# 配置服务端口为 8080
$ vi /etc/nginx/nginx.conf

$ systemctl enable nginx --now

准备 OpenShift Nodes 部署所需的配置文件

OpenShift 的部署期间需要用到多个文件:

  • 安装配置文件:install-config.yaml
  • Kubernetes 部署清单:Manifest
  • Ignition 配置文件(包含了 Machine Types):该文件描述了如何创建 OpenShift Cluster。

install-config.yaml 将被转换为 Manifest 文件,然后再将 Manifest 文件包装到 Ignition 配置文件中。最终,安装程序使用这些 Ignition 配置文件来创建 Openshift 集群。运行安装程序时,所有原始安装配置文件都会修改,因此在安装之前应该先备份文件。

NOTE:安装程序生成的 Ignition 配置文件包含 24 小时后过期的证书,所以必须在证书过期之前完成集群安装。

$ mkdir /okdinstall

# 根据需要修改 OKD 部署配置
$ vi /okdinstall/install-config.yaml

apiVersion: v1
baseDomain: example.com         # 配置基础域名,OpenShift 内部所有的 DNS 记录必须是此基础域名的子域,并包含集群名称。
compute:
- hyperthreading: Enabled 
  name: worker
  replicas: 0                   # 配置 Worker Node 数量,因为我们要手动创建 Worker Node,所以这里设置为 0。
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 1                   # 配置 Master Node 数量,我们部署单 Master,所以写 1,注意:Master Node 数量必须和 etcd 节点数量一致。
metadata:
  name: okd                     # 集群名称。
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14         # 配置 Pod IP Pool。注意:Pod IP Pool 不能与物理网络冲突。
    hostPrefix: 23              # 分配给每个节点的子网前缀长度。
  networkType: OVNKubernetes    # 配置集群网络类型
  serviceNetwork: 
  - 172.30.0.0/16               # 配置 Service IP Pool。
platform:
  none: {}                      # 平台类型,因为我们使用裸金属安装类别,所有不填。
fips: false

# 配置 Bastion SSH pub key
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD1zXmIde90FOxXQIgNZgECZux0pwwMQ3CpRnvYozTqIP6jI//VNZJq0rKyOVfq8hvzYtJDGqJvUJwwXPS3jaI1X0eajRYCIEJ6fu71nLGhAd8igPp4AsP9TFzyIpReQXuGL990PJl4uFtGhCcr3PMLbLNC5qTt+rDKdYWexD74IpOZe3z14hRgHitcBsPmoKeMxTJlr+dYDW5n3tyBw/4cOpcdTSQVO4SWlmDzjrmCDGh13J7C60BB2PrLnHkrYjud66Pwcr6HcSTjq8Y/amx5aI+hp7wq2iWInGYsPW/6REIOm83pos5s+x9SOkM17TW/qmbqKivngyW5qNzXgzJnfiEg4YEGMoTcPfvYzaonRYhe01nf887eXZi5bemJoGyuTE92Ex/hokLxaZNVWIMxBLEVh9BJiDGALqvuPDmoDJIkIpqX0E2gL1Nz0iYZogRkdifnZpfVUrww9PS0xrUd0AwiFDn6Ng09qx8K8lJ/o2RgQJvVCqOeeXXrRWtdv6zS9IirmiqmQT5w8ywnQ2yi6wXrWG2BxMSJgV6pr2mwsg5GTEPMoM4PlUBExM8ZmlbvbDzqauZ2bDoFZGGFs4LqImf0zdS7nRx81q3i96Qqx8UCuArh/81YefLpLtYcl95e+7M9E8/mXTf0t9yeZYkDw0Iskn9TXMQ1JCW77Xvhvw== root@bastion.okd.example.com'

# 配置 Image Registry 地址
imageContentSources:
- mirrors:
  - registry.okd.example.com:5000/openshift/okd
  source: quay.io/openshift/okd
- mirrors:
  - registry.okd.example.com:5000/openshift/okd
  source: quay.io/openshift/okd-content

# 配置 Image Registry 的密钥信息
pullSecret: '{"auths":{"registry.okd.example.com:5000": {"auth": "YWRtaW46YWRtaW4=","email": ""}}}'

# 配置 Image Registry 的信任证书。
# cat /opt/registry/certs/domain.crt
# 注意:前面要保持两个空格作为缩进。
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIDbTCCAlWgAwIBAgIJAP6IHiEew+eNMA0GCSqGSIb3DQEBCwUAME0xHzAdBgNV
  BAMMFnJlZ2lzdHJ5Lm9rZC5pbnRlbC5jb20xHTAbBgNVBAoMFE15IENvbXBhbnkg
  TmFtZSBMVEQuMQswCQYDVQQGEwJVUzAeFw0yMDEyMjIxMTUwMzVaFw0yMTEyMjIx
  MTUwMzVaME0xHzAdBgNVBAMMFnJlZ2lzdHJ5Lm9rZC5pbnRlbC5jb20xHTAbBgNV
  BAoMFE15IENvbXBhbnkgTmFtZSBMVEQuMQswCQYDVQQGEwJVUzCCASIwDQYJKoZI
  hvcNAQEBBQADggEPADCCAQoCggEBAN4jbffKnGVz8U6gcWIA/ug5kQH1lVtaNmcS
  sHVeqLwjfVlTSkK+yyhY4AyN6YWWhznaM9zqb9ffQXyP1zuwj2UgYfiZCFD4MDEd
  vHbMYLjkQ3w4LivmW/+4zZJmn8LZsQp4RpfliBDT2bc6ZuXnbL3Z6cF++/pA935J
  pxFHdmKekfa+foKZrEx2u+7F9JazrfkSJVsc3lPmqVyCBK4Eak0VWcxInAUh+ajh
  MCOwY+bepDpvpZLal2OVQBt8XiE0Aw0prhWtHbhLBgNMPhUf5idQYOuJKM0JAoAf
  GPtWXF1WZJ18gsMY6JiKLQg3AM5qVTPgNjNBL/XecY+QyihyFScCAwEAAaNQME4w
  HQYDVR0OBBYEFG6+XH3QEWpHvDUp32m9zaomGnH4MB8GA1UdIwQYMBaAFG6+XH3Q
  EWpHvDUp32m9zaomGnH4MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEB
  AGN5XTlZEvvxNdt/PyLAb+K0RK2kkitfM6c4Von7+QZSqerrtXGPce+NqIrCD7Me
  Gp9gmExlOLuVcduwEX1BdkAgASQEDtWBWR3lXO0pgM8LDQuSI59ztuoiuIYi6pFP
  wrjLND5R/15TMpc2mlFUz2Jv484HTmfQPJVAvdwGDGTJEom4mrytmSEBq0v8Gtdc
  A9Fi6XQZj9PW98kdPi+Emhbp9srNIBnyP/NzXGLz90jwmFWxPfLnwJ1wPNYqvw30
  i+B+jcfvVTpmX5JJTmPXGLNLgrROUjOE0S45BJFYim1ACu7ZDAluFlxHdOUAylzR
  TJcpgeuhQ/Aoj+SbFr0BjEg=
  -----END CERTIFICATE-----

NOTE:建议先备份

$ cp /okdinstall/install-config.yaml /tmp/install-config.yaml.bak 
  • 生产 manifests 部署 YAML 文件:
$ openshift-install create manifests --dir=/okdinstall

INFO Consuming Install Config from target directory 
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings 
WARNING Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated 
  • 编辑 manifests/cluster-scheduler-02-config.yml 文件,将 mastersSchedulable 的值设为 flase,以防止将 Pod 调度到 Master Node:
$ sed -i 's/mastersSchedulable: true/mastersSchedulable: False/' /okdinstall/manifests/cluster-scheduler-02-config.yml
  • 创建 Ignition 文件,作为 OpenShift Nodes 的配置文件:
$ openshift-install create ignition-configs --dir=/okdinstall

# 生成的文件目录树
├── auth
│   ├── kubeadmin-password
│   └── kubeconfig
├── bootstrap.ign
├── master.ign
├── metadata.json
└── worker.ign


NOTE:其中,auth 目录下是 Kubernetes 的认证信息,拷贝到环境后才能使用 oc 或 kubectl 指令:

$ mkdir /root/.kube/

$ cp /okdinstall/auth/kubeconfig ~/.kube/config
  • 将 Ignition 文件拷贝到 Nginx 中,供各个节点在执行安装时进行下载:
$ mkdir /usr/share/nginx/html/ignition

$ chmod -R 755 /okdinstall/*

$ cp -rp /okdinstall/* /usr/share/nginx/html/ignition/
  • 下载 Fedora CoreOS 镜像文件并拷贝到 Nginx 中,供各个节点在执行安装时进行下载:
$ mkdir /usr/share/nginx/html/install

$ wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200907.3.0/x86_64/fedora-coreos-32.20200907.3.0-metal.x86_64.raw.xz

$ mv fedora-coreos-32.20200907.3.0-metal.x86_64.raw.xz /usr/share/nginx/html/install/coreos.raw.xz

部署 Bootstrap Node(引导节点)

OpenShift 集群部署采用 “通过 Kubernetes 来安装升级 Kubernetes” 的思路,所以首先会在 Bootstrap 启动一套 “Micro-Kubernetes” 来部署 OpenShift Cluster。

Bootstrap Node 应该可以访问所有 Nodes,它会启动一个临时的 Kubernetes Control Plane(控制平面),以此来完成 OpenShift Cluster 的部署。

在自动化引导部署的场景中,Bootstrap Node 会完成以下步骤:

  1. Bootstrap Node 启动并开始托管 Master Node 启动所需的资源。
  2. Master Node 从 Bootstrap Node 远程获取资源并完成引导。
  3. Master Node 通过 Bootstrap Node 构建 ETCD 集群。
  4. Bootstrap Node 使用 ETCD 集群启动临时 Kubernetes Control Plane。
  5. 临时控制平面在 Master Node 启动生产控制平面。
  6. 临时控制平面关闭并将控制权传递给生产控制平面。
  7. Bootstrap Node 将 OpenShift Cluster 组件注入生产控制平面。
  8. 安装程序关闭 Bootstrap Node。
  9. 引导安装过程完成以后,OpenShift Cluster 部署完毕。然后集群开始下载并配置日常操作所需的其余组件,包括创建计算节点、通过 Operator 安装其他服务等。
  10. 集群部署完成后可删除 Bootstrap Node。

因为,我们采用的是 “半自动引导部署” 的方式,所以需要从 U 盘启动并部署 Bootstrap Node。进入到 Kernel 选择页面后,按下 tab 键,开始填写引导信息:

ip=192.168.120.102::192.168.120.101:255.255.255.0:bootstrap.okd.example.com:ens787f0:none nameserver=192.168.120.101 coreos.inst.install_dev=sda coreos.inst.image_url=http://192.168.120.101:8080/install/coreos.raw.xz coreos.inst.ignition_url=http://192.168.120.101:8080/ignition/bootstrap.ign

注:引导信息分为以下几个部分:

  • IP 地址
  • 网关地址
  • 地址掩码
  • 主机名
  • 管理网网卡名
  • 静态 IP 地址类型
  • DNS 主机地址
  • 安装磁盘名
  • image 文件源
  • ignition 文件源

如下图:
在这里插入图片描述

  • 检查部署是否完成:
$ ssh -i ~/.ssh/id_rsa core@bootstrap.okd.example.com

# 可以看到 7 个 Ready 的 Pods
$ crictl pods

POD ID              CREATED             STATE               NAME                                                                  NAMESPACE                             ATTEMPT
17a978b9e7b1e       3 minutes ago       Ready               bootstrap-kube-apiserver-bootstrap.openshift4.example.com             kube-system                           24
8a0f79f38787a       3 minutes ago       Ready               bootstrap-kube-scheduler-bootstrap.openshift4.example.com             kube-system                           4
1a707da797173       3 minutes ago       Ready               bootstrap-kube-controller-manager-bootstrap.openshift4.example.com    kube-system                           4
0461d2caa2753       3 minutes ago       Ready               cloud-credential-operator-bootstrap.openshift4.example.com            openshift-cloud-credential-operator   4
ab6519286f65a       3 minutes ago       Ready               bootstrap-cluster-version-operator-bootstrap.openshift4.example.com   openshift-cluster-version             2
457a7a46ec486       8 hours ago         Ready               bootstrap-machine-config-operator-bootstrap.openshift4.example.com    default                               0
e4df49b4d36a1       8 hours ago         Ready               etcd-bootstrap-member-bootstrap.openshift4.example.com                openshift-etcd                        0

NOTE:因为我们采用的是 One Master 部署方式,所以需要在 Bastion Node 打上 One ETCD 的 Patch。Master 和 ETCD 节点数量必须保持一致。

# 编辑文件,写入内容。必须打这个patch,不然直接改副本数,还会恢复回去。

$ vi /opt/etcd_quorum_guard.yaml
- op: add
  path: /spec/overrides
  value:
  - kind: Deployment
    group: apps/v1
    name: etcd-quorum-guard
    namespace: openshift-machine-config-operator
    unmanaged: true


$ oc patch clusterversion version --type json -p "$(cat /opt/etcd_quorum_guard.yaml)"

$ oc scale --replicas=1 deployment/etcd-quorum-guard -n openshift-machine-config-operator

在安装 Master Node 之前不妨持续观察一下 Bootstrap Node 的日志:

$ journalctl -b -f -u bootkube.service

部署 Master Node

同样的,使用 U 盘引导,引导参数如下:

ip=192.168.120.103::192.168.120.101:255.255.255.0:master.okd.example.com:ens787f0:none nameserver=192.168.120.101 coreos.inst.install_dev=sda coreos.inst.image_url=http://192.168.120.101:8080/install/coreos.raw.xz coreos.inst.ignition_url=http://192.168.120.101:8080/ignition/master.ign

Master Node 安装成功后会重启一次,然后同样可以从 Bastion Node 上通过 SSH 密钥登录。

  • 检查部署是否成功,在 Bastion Node 上执行:
$ openshift-install --dir=/okdinstall wait-for bootstrap-complete --log-level=debug

DEBUG OpenShift Installer 4.5.0-0.okd-2020-07-14-153706-ga 
DEBUG Built from commit 290e3b1de6096ecef2133fb071ff3a71c9c78594 
INFO Waiting up to 20m0s for the Kubernetes API at https://api.okd.example.com:6443... 
INFO API v1.18.3 up                               
INFO Waiting up to 40m0s for bootstrapping to complete... 
DEBUG Bootstrap status: complete                   
INFO It is now safe to remove the bootstrap resources 
DEBUG Time elapsed per stage:                      
DEBUG Bootstrap Complete: 5s                       
DEBUG                API: 5s                       
INFO Time elapsed: 5s   
  • 如果希望再 Master Node 上执行 oc 指令,还需要从 Bastion Node 上把 auth 目录拷贝到过来:
scp -rp /root/.kube/ core@master:/root/.kube

NOTE:至此,Bootstrap Node 的任务就完成了,你可以选择删除掉它,或者修改 Bastion Node 上的 HAProxy 配置文件,注释掉其中的 backend bootstrap 的部分。

部署 Worker Node

同样的,使用 U 盘引导,引导参数如下:

ip=192.168.120.104::192.168.120.101:255.255.255.0:worker1.okd.example.com:ens786f1:none nameserver=192.168.120.101 coreos.inst.install_dev=sda coreos.inst.image_url=http://192.168.120.101:8080/install/coreos.raw.xz coreos.inst.ignition_url=http://192.168.120.101:8080/ignition/worker.ign
  • 检查部署是否成功,在 Bastion Node 上执行:
$ openshift-install --dir=/okdinstall wait-for install-complete --log-level debug

DEBUG OpenShift Installer 4.5.0-0.okd-2020-07-14-153706-ga 
DEBUG Built from commit 290e3b1de6096ecef2133fb071ff3a71c9c78594 
DEBUG Fetching Install Config...                   
DEBUG Loading Install Config...                    
DEBUG   Loading SSH Key...                         
DEBUG   Loading Base Domain...                     
DEBUG     Loading Platform...                      
DEBUG   Loading Cluster Name...                    
DEBUG     Loading Base Domain...                   
DEBUG     Loading Platform...                      
DEBUG   Loading Pull Secret...                     
DEBUG   Loading Platform...                        
DEBUG Using Install Config loaded from state file  
DEBUG Reusing previously-fetched Install Config    
INFO Waiting up to 30m0s for the cluster at https://api.okd.example.com:6443 to initialize... 
DEBUG Still waiting for the cluster to initialize: Working towards 4.5.0-0.okd-2020-07-14-153706-ga: 88% complete 
DEBUG Cluster is initialized                       
INFO Waiting up to 10m0s for the openshift-console route to be created... 
DEBUG Route found in openshift-console namespace: console 
DEBUG Route found in openshift-console namespace: downloads 
DEBUG OpenShift console route is created           
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/okdinstall/auth/kubeconfig' 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.okd.example.com 
INFO Login to the console with user: "kubeadmin", and password: "YCEWn-iqq56-fPFT5-jKbh8" 
DEBUG Time elapsed per stage:                      
DEBUG Cluster Operators: 18s                       
INFO Time elapsed: 18s     
  • 部署完后在 Bastion Node 上批准 Worker Node 的加入:将 Worker Node 添加到 Cluster 时,会为添加的每台节点生成两个待处理证书签名请求(CSR)。必须确认这些 CSR 已获得批准,或者在必要时自行批准。
# 查看挂起的证书签名请求(CSR),并确保添加到集群的每台节点都能看到具有 Pending 或 Approved 状态的客户端和服务端请求。针对 Pending 状态的 CSR 批准请求执行批准:$ oc adm certificate approve xxx
$ oc get csr

# 或者执行以下命令批准所有 CSR:
$ oc get csr -o json | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

NOTE:因为我们采用了 One Worker Node 部署方式,所以需要调整副本数量。

# 修改下面这些服务副本数为 1。

$ oc scale --replicas=1 ingresscontroller/default -n openshift-ingress-operator
$ oc scale --replicas=1 deployment.apps/console -n openshift-console
$ oc scale --replicas=1 deployment.apps/downloads -n openshift-console
$ oc scale --replicas=1 deployment.apps/oauth-openshift -n openshift-authentication
$ oc scale --replicas=1 deployment.apps/packageserver -n openshift-operator-lifecycle-manager

$ oc scale --replicas=1 deployment.apps/prometheus-adapter -n openshift-monitoring
$ oc scale --replicas=1 deployment.apps/thanos-querier -n openshift-monitoring
$ oc scale --replicas=1 statefulset.apps/prometheus-k8s -n openshift-monitoring
$ oc scale --replicas=1 statefulset.apps/alertmanager-main -n openshift-monitoring

访问 Dashboard

INFO Access the OpenShift web-console here: https://console-openshift-console.apps.okd.example.com 
INFO Login to the console with user: "kubeadmin", and password: "YCEWn-iqq56-fPFT5-jKbh8" 

浏览器访问 https://console-openshift-console.apps.okd.example.com,输入账户名/密码。

Troubleshooting

Worker Node 镜像认证不通过

$ /usr/bin/podman pull --authfile=/var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:e4d51b905391ff610e4082c5185c5598b2d9e81aec1f1436d56636404d4a0717 
Trying to pull quay.io/openshift/okd-content@sha256:e4d51b905391ff610e4082c5185c5598b2d9e81aec1f1436d56636404d4a0717...

  Get "https://quay.io/v2/": dial tcp: lookup quay.io on 192.168.120.101:53: server misbehaving
Error: unable to pull quay.io/openshift/okd-content@sha256:e4d51b905391ff610e4082c5185c5598b2d9e81aec1f1436d56636404d4a0717: Error initializing source docker://quay.io/openshift/okd-content@sha256:e4d51b905391ff610e4082c5185c5598b2d9e81aec1f1436d56636404d4a0717: (Mirrors also failed: [registry.okd.example.com:5000/openshift/okd@sha256:e4d51b905391ff610e4082c5185c5598b2d9e81aec1f1436d56636404d4a0717: error pinging docker registry registry.okd.example.com:5000: Get "https://registry.okd.example.com:5000/v2/": x509: certificate signed by unknown authority]): quay.io/openshift/okd-content@sha256:e4d51b905391ff610e4082c5185c5598b2d9e81aec1f1436d56636404d4a0717: error pinging docker registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 192.168.120.101:53: server misbehaving


解决

# 来自 bootstrap 节点 cat /etc/pki/ca-trust/source/anchors/ca.crt

$ vi /etc/pki/ca-trust/source/anchors/ca.crt
-----BEGIN CERTIFICATE-----
MIIDbTCCAlWgAwIBAgIJAP6IHiEew+eNMA0GCSqGSIb3DQEBCwUAME0xHzAdBgNV
BAMMFnJlZ2lzdHJ5Lm9rZC5pbnRlbC5jb20xHTAbBgNVBAoMFE15IENvbXBhbnkg
TmFtZSBMVEQuMQswCQYDVQQGEwJVUzAeFw0yMDEyMjIxMTUwMzVaFw0yMTEyMjIx
MTUwMzVaME0xHzAdBgNVBAMMFnJlZ2lzdHJ5Lm9rZC5pbnRlbC5jb20xHTAbBgNV
BAoMFE15IENvbXBhbnkgTmFtZSBMVEQuMQswCQYDVQQGEwJVUzCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAN4jbffKnGVz8U6gcWIA/ug5kQH1lVtaNmcS
sHVeqLwjfVlTSkK+yyhY4AyN6YWWhznaM9zqb9ffQXyP1zuwj2UgYfiZCFD4MDEd
vHbMYLjkQ3w4LivmW/+4zZJmn8LZsQp4RpfliBDT2bc6ZuXnbL3Z6cF++/pA935J
pxFHdmKekfa+foKZrEx2u+7F9JazrfkSJVsc3lPmqVyCBK4Eak0VWcxInAUh+ajh
MCOwY+bepDpvpZLal2OVQBt8XiE0Aw0prhWtHbhLBgNMPhUf5idQYOuJKM0JAoAf
GPtWXF1WZJ18gsMY6JiKLQg3AM5qVTPgNjNBL/XecY+QyihyFScCAwEAAaNQME4w
HQYDVR0OBBYEFG6+XH3QEWpHvDUp32m9zaomGnH4MB8GA1UdIwQYMBaAFG6+XH3Q
EWpHvDUp32m9zaomGnH4MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEB
AGN5XTlZEvvxNdt/PyLAb+K0RK2kkitfM6c4Von7+QZSqerrtXGPce+NqIrCD7Me
Gp9gmExlOLuVcduwEX1BdkAgASQEDtWBWR3lXO0pgM8LDQuSI59ztuoiuIYi6pFP
wrjLND5R/15TMpc2mlFUz2Jv484HTmfQPJVAvdwGDGTJEom4mrytmSEBq0v8Gtdc
A9Fi6XQZj9PW98kdPi+Emhbp9srNIBnyP/NzXGLz90jwmFWxPfLnwJ1wPNYqvw30
i+B+jcfvVTpmX5JJTmPXGLNLgrROUjOE0S45BJFYim1ACu7ZDAluFlxHdOUAylzR
TJcpgeuhQ/Aoj+SbFr0BjEg=
-----END CERTIFICATE-----

$ update-ca-trust extract

修复

$ /usr/bin/podman pull --authfile=/var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:e4d51b905391ff610e4082c5185c5598b2d9e81aec1f1436d56636404d4a0717 
Trying to pull quay.io/openshift/okd-content@sha256:e4d51b905391ff610e4082c5185c5598b2d9e81aec1f1436d56636404d4a0717...
Getting image source signatures
Copying blob 5f87fb0c9624 done  
Copying blob 15ca123cecfd [===================>------------------] 26.9MiB / 51.8MiB
Copying blob bb0da44cdbce done  
Copying blob 309f2519e870 done  
Copying blob a03401a44180 [==============>-----------------------] 28.2MiB / 72.7MiB

已标记关键词 清除标记
©️2020 CSDN 皮肤主题: 博客之星2020 设计师:CY__ 返回首页
实付 49.90元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值