okd4.5 install fail

[xiaosayishi]

When I installed okd4.5 on centos7.9, an error was reported when I executed the openshift command

[root@bastion ~]# openshift-install --dir=/okdinstall/ wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.5.0-0.okd-2020-10-15-235428

DEBUG Built from commit 63200c80c431b8dbaa06c0cc13282d819bd7e5f8
INFO Waiting up to 20m0s for the Kubernetes API at https://api.okd.test1.com:6443...
DEBUG Still waiting for the Kubernetes API: Get https://api.okd.test1.com:6443/version?timeout=32s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-apiserver-lb-signer")

I don’t understand why this problem occurs. I ask you all for help. Thank you.

The following are my installation steps：

52 hostnamectl set-hostname bastion
53 ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/id_rsa

54 eval "$(ssh-agent -s)"

55 ssh-add ~/.ssh/id_rsa

56 systemctl disable firewalld

57 systemctl stop firewalld

58 yum -y install wget vim etcd bind-utils

59 systemctl enable etcd --now

60 systemctl status etcd

61 wget https://github.com/coredns/coredns/releases/download/v1.6.9/coredns_1.6.9_linux_amd64.tgz

62 tar zxvf coredns_1.6.9_linux_amd64.tgz

63 mv coredns /usr/local/bin

64 useradd coredns -s /sbin/nologin

65 vim /etc/systemd/system/coredns.service

66 mkdir /etc/coredns

67 vi /etc/coredns/Corefile

68 systemctl enable coredns --now

69 systemctl status coredns

70 alias etcdctlv3='ETCDCTL_API=3 etcdctl'

71 etcdctlv3 put /skydns/com/test1/okd/api '{"host":"172.16.121.13", "ttl":60}'

72 etcdctlv3 put /skydns/com/test1/okd/api-int '{"host":"172.16.121.13", "ttl":60}'

73 etcdctlv3 put /skydns/com/test1/okd/registry '{"host":"172.16.121.13", "ttl":60}'

74 etcdctlv3 put /skydns/com/test1/okd/etcd-1 '{"host":"172.16.121.14", "ttl":60}'

75 etcdctlv3 put /skydns/com/test1/okd/etcd-2 '{"host":"172.16.121.15", "ttl":60}'

76 etcdctlv3 put /skydns/com/test1/okd/etcd-3 '{"host":"172.16.121.16", "ttl":60}'

77 etcdctlv3 put /skydns/com/test1/okd/_tcp/_etcd-server-ssl/x1 '{"host":"etcd-1.okd.test1.com", "ttl":60, "priority":0, "weight":10, "port":2380}'

78 etcdctlv3 put /skydns/com/test1/okd/_tcp/_etcd-server-ssl/x1 '{"host":"etcd-2.okd.test1.com", "ttl":60, "priority":0, "weight":10, "port":2380}'

79 etcdctlv3 put /skydns/com/test1/okd/_tcp/_etcd-server-ssl/x1 '{"host":"etcd-3.okd.test1.com", "ttl":60, "priority":0, "weight":10, "port":2380}'

80 etcdctlv3 put /skydns/com/test1/okd/bastion '{"host":"172.16.121.13", "ttl":60}'

81 etcdctlv3 put /skydns/com/test1/okd/bootstrap '{"host":"172.16.121.19", "ttl":60}'

82 etcdctlv3 put /skydns/com/test1/okd/master1 '{"host":"172.16.121.14", "ttl":60}'

83 etcdctlv3 put /skydns/com/test1/okd/master2 '{"host":"172.16.121.15", "ttl":60}'

84 etcdctlv3 put /skydns/com/test1/okd/master3 '{"host":"172.16.121.16", "ttl":60}'

85 etcdctlv3 put /skydns/com/test1/okd/worker1 '{"host":"172.16.121.17", "ttl":60}'

86 etcdctlv3 put /skydns/com/test1/okd/worker2 '{"host":"172.16.121.18", "ttl":60}'

87 dig +short apps.okd.test1.com @127.0.0.1

88 dig +short master2.okd.test1.com @127.0.0.1

89 cat /etc/resolv.conf

 search okd.test1.com  
 nameserver 172.16.121.13

90 yum -y install haproxy

91 vim /etc/haproxy/haproxy.cfg

92 systemctl enable haproxy && systemctl restart haproxy

93 systemctl status haproxy

94 netstat -anput | grep 6443

95 yum -y install net-tools

96 netstat -anput | grep 6443

97 netstat -anput | grep 22623

98 mkdir -p /opt/registry/{auth,certs,data}

99 mkdir /data

100 cd /opt/registry/certs

101 openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 36500 -out ca.crt

102 openssl req -newkey rsa:4096 -nodes -sha256 -keyout registry.okd.test1.com.key -out registry.okd.test1.com.csr -days 36500

103 openssl x509 -req -days 36500 -in registry.okd.test1.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out registry.okd.test1.com.crt

104 echo -n 'admin:Harbor12345' | base64 -w0

105 vim /root/pull-secret.json

106 yum -y install httpd-tools yum-utils telnet httpd epel-release podman

107 yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

108 yum -y install docker-ce docker-compose

109 systemctl start docker && systemctl enable docker

110 systemctl status docker

111 cd /opt

112 wget https://github.com/goharbor/harbor/releases/download/v2.0.1/harbor-offline-installer-v2.0.1.tgz

121 tar -zxvf harbor-offline-installer-v2.0.1.tgz && rm -rf harbor-offline-installer-v2.0.1.tgz && cd harbor/

122 mv harbor.yml.tmpl harbor.yml

123 vim harbor.yml

124 ./install.sh

125 mkdir -p /etc/docker/certs.d/registry.okd.test1.com

126 cp /opt/registry/certs/registry.okd.test1.com.crt /etc/docker/certs.d/registry.okd.test1.com
127 cp /opt/registry/certs/ca.crt /etc/pki/ca-trust/source/anchors/

128 update-ca-trust extrat

129 systemctl restart docker

130 netstat -anput | grep docker

131 curl -u admin:Harbor12345 -k https://registry.okd.test1.com:18443/v2/_catalog

132 wget https://github.com/openshift/okd/releases/download/4.5.0-0.okd-2020-10-15-235428/openshift-client-linux-4.5.0-0.okd-2020-10-15-235428.tar.gz

133 ll

134 cd /root/

135 ll

136 wget https://github.com/openshift/okd/releases/download/4.5.0-0.okd-2020-10-15-235428/openshift-install-linux-4.5.0-0.okd-2020-10-15-235428.tar.gz

137 ll

138 tar -zxvf openshift-client-linux-4.5.0-0.okd-2020-10-15-235428.tar.gz

139 cp oc kubectl /usr/local/bin/

140 oc version

141 tar -zxvf openshift-install-linux-4.5.0-0.okd-2020-10-15-235428.tar.gz

142 cp openshift-install /usr/local/bin/

143 openshift-install version

144 export OCP_RELEASE=4.5.0-0.okd

145 export LOCAL_REGISTRY='registry.okd.test1.com:18443'

146 export LOCAL_REPOSITORY='openshift/okd'

147 export PRODUCT_REPO='openshift'

148 export LOCAL_SECRET_JSON='/root/pull-secret.json'

149 export RELEASE_NAME='okd'

150 export ARCHITECTURE="2020-10-15-235428"

152 oc adm -a ${LOCAL_SECRET_JSON} release mirror --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

209 cd /root/

210 vim install-config.yaml

211 mkdir /okdinstall/

213 cp install-config.yaml /okdinstall/

214 openshift-install create manifests --dir=/okdinstall

215 sed -i 's/mastersSchedulable: true/mastersSchedulable: False/'
/okdinstall/manifests/cluster-scheduler-02-config.yml

216 openshift-install create ignition-configs --dir=/okdinstall

217 mkdir /root/.kube/

218 cp /okdinstall/auth/kubeconfig ~/.kube/config

219 chmod -R 755 /okdinstall/*

220 mkdir /usr/share/nginx/html/ignition/

221 cp -rp /okdinstall/* /usr/share/nginx/html/ignition/

222 mkdir /usr/share/nginx/html/install/

223 openshift-install --dir=/okdinstall/ wait-for bootstrap-complete --log-level=debug

设计到的文件内容如下： cat /root/pull-secret.json
{
"auths": {
"registry.okd.test1.com:18443": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU=",
"email": ""
}
}
}
cat /root/install-config.yaml
apiVersion: v1
baseDomain: test1.com
compute:

• hyperthreading: Enabled
  name: worker
  replicas: 0
  controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
  metadata:
  name: okd
  networking:
  clusterNetwork:
  • cidr: 10.128.0.0/14
    hostPrefix: 23
    networkType: OpenShiftSDN
    serviceNetwork:
  • 172.30.0.0/16
    platform:
    none: {}
    fips: false

sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3EhKr+wP22gSg3yoahLDR2cNxHEfm59/uFc4XBFdROEXofS4pm3V3c3ug2dRCtaVI/Wb2wUKjvbJd3Cw8dHhD4cU6rIVBt2+P6GpOIslkFb59RjUdWhS7EVcFNg/5AJJc4gmg+Read1WmJQny36Ogpf8WfMXyCeIMB1w7cezIg/E1lJo0LonSWfj4vUUweqZzWhfBgiet6NxyRm6C8ahWMhoMvaVNd2JK4RtMxP2vdX9HpEjUQNFnPG0N4wcJztIKLjh5IUPMNrTwFe62zB6n3kx6ZycrZUQmRHJVss3QN566bxNu2qmbfryvktj2lGFon8pt08Fwe5gJcHNbLhKTmQZc1KEh9leKXiTFcE/omsA3fZC4K+u3Xt69NHBLVTaBA9t4/gETvU7knQO51ffXvS7ut61FfMObuU9e1E7Vzy4czyFNrv4yRIUv6tdj35zQd96hvNJeNPx4Owogbc/5tWhkc0fPgDZCn5v+DIh8yJZ2GL0nrCI1eS0eIHyWuqMY7PbrQRSp3bQLomEvlLB+Eg/lyv0nhVgbue6bgrWy6fN2ExEJHBn2rwZOq9j/DiU2xF+AdALQNeqVHd6GBo8Q/TC3w6TVlemLDnAyrf7JSQNUIVstRyYNkqN/3/is9ibPQqYckEIvDenonuBUe1Np7DH3qcSkjAFz3xdQ== root@bastion'

pullSecret: '{"auths":{"registry.okd.test1.com:18443": {"auth": "YWRtaW46SGFyYm9yMTIzNDU=","email": ""}}}'
imageContentSources:

• mirrors:
  • registry.okd.test1.com:18443/openshift/okd
    source: quay.io/openshift/okd
• mirrors:
  • registry.okd.test1.com:18443/openshift/okd
    source: quay.io/openshift/okd-content

密钥文件信息来自于 /etc/docker/certs.d/registry.okd.test1.com/registry.okd.test1.com.crt

additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
MIIFejCCA2ICCQCIsY5LihXiSDANBgkqhkiG9w0BAQsFADB+MQswCQYDVQQGEwJD
TjELMAkGA1UECAwCQkoxCzAJBgNVBAcMAkJKMQ0wCwYDVQQKDARURVNUMQ0wCwYD
VQQLDARURVNUMRgwFgYDVQQDDA8qLm9rZC50ZXN0MS5jb20xHTAbBgkqhkiG9w0B
CQEWDnRlc3RAdGVzdDEuY29tMCAXDTIxMDgxMDExMDc0MloYDzIxMjEwNzE3MTEw
NzQyWjB+MQswCQYDVQQGEwJDTjELMAkGA1UECAwCQkoxCzAJBgNVBAcMAkJKMQ0w
CwYDVQQKDARURVNUMQ0wCwYDVQQLDARURVNUMRgwFgYDVQQDDA8qLm9rZC50ZXN0
MS5jb20xHTAbBgkqhkiG9w0BCQEWDnRlc3RAdGVzdDEuY29tMIICIjANBgkqhkiG
9w0BAQEFAAOCAg8AMIICCgKCAgEA2PpMgCdjfZd2ww+uYjxrN3Wbl5qxKWJYTtX9
uI0/37g1Bz09XSVB4zW3WHmqCBwr1ad6/S92vv0HHtnJDP+PcoP6FsnBagPxxwCC
bIF4svkwSG7bQkUZcFbQ+S3Tqo8cHBOeSW4fzFPYO7QKF3Bhmq1tqYO5cqPrlOdb
1+L31Etx+P6P7+/XBzr4CUckTsdQXoEIcKFTQfrn9NMAeSscFk5Qw76DPhj4tJbe
Rx+ebpwgQodrsr8C9qn8Ebj6n1oz5S/mer6CeNR4M2HTdoBqUP0eHbLAyBzrzXwK
0OTqn7+El5DyA86yY99ag8SLsNjHc1J4PYPLbNuk/UvathsLAWQZjeyq+Zyf4KQk
PvXi8wlKXik9eAAtL/fx3jNoAm57f0qdc7Z3+e8X4cr9wvNerC7IVV8OSRMsxlze
w5c0YqP3kex9UV3f4FpKBJEZI9k1Gcg911ffl5dj7wDunj2wzyzprhjWvdswkCux
5kJfPZhgoBUK0DRmoW7b9r00YsMYodI05MuEVG/KiRlfQn3/xa0NEAqxJbM+zbOJ
rJmQLV2mJQjh3FgIc99L0CFPaoaANPi1t3hYnFRiWxRaDhxnNUaCb6QUMp+ZphbX
eRYmZ2DqGB91pFliT8MSxlBqcTq+AgUE7IKa0Tu9QZxN9ve1DlXJQi+LIenC3iZs
7OMQKwUCAwEAATANBgkqhkiG9w0BAQsFAAOCAgEAZZ/8l4daDzyxwIALkoepjbKi
ozIp/Oz9HRvlVUM1rM37zceo3xumP8T1AWvr/z9tLei5/HMgqAbbegFx+Y0Rdwyt
DdHk4UFOcGBomFnA0CYXOywKIOo5qByOthBu0GF7quns4MTpPUN0sXkYDAoBRPLB
Wsan71DBAa6GN/kvu5eXN3SZeiAz4Ou3Yk43++Iy5NNWNjB37GNdtL9LRdtYGaVb
/kUiFdPL2S9gtex/EdWFqsZrN/0VNGRD8/bLQlavIFwYYz7zaMt7UxT7q7v2prkK
FF7GZiDJZb0NeHDLnyoP0yHuqefnKV7oV1aBqYh3Y6zk16S6DIl8WKfCgEaHSwB+
ESGxKU42NCWgTw8ollXHMjezT1C6LAzLGuMToJXVQzYhoacGLJKtkCcKwYV/df4z
/Z5s1ECphLLetR8JyByhDOzk48tZyXrMN3DvtIfVsbR5cRqZrW5KK6gRKTm1Stsq
TLB9wLtn/ytA9HaMijsudPB/7k/Q3Mm4G3WfnLlpQpc9oQeG/32dEq2FM3Ykp/YX
UQvTh6MEjj8XTHe6FT1yJNMqcQL/oct3bTRc8kegtzgELgpUGMGBnlFtIICBleVx
2/lzFrU7P08poKuZ+1loG4rZG9vxsWRX/MZ2AVi+c2jQ9qRwnD5as+YoUyFw42Xx
vOouHXd0lg7onR1v82M=
-----END CERTIFICATE-----

[vrutkovs]

Please attach log bundle

Is this reproducible on latest OKD 4.7?

[xiaosayishi]

I used OKD 4.5

[xiaosayishi]

Hi，vrutkovs：
I haven't installed the cluster, or even bootstart. So the current situation has not produced log information. But I got this error message when I executed the command. Normally it should "DEBUG Still waiting for the Kubernetes API: Get https://api.okd.test1.com:6443/version?timeout=32s EOF".
I just made preparations before installing the cluster, and I don’t know why this error message appears

[rvanderp3]

Here are some things to check:

• Are the bootstrap and master nodes powered on?
• Are the nodes showing any errors? For example, are they failing to retrieve an ignition? For the master nodes, this is normal until the bootstrap node is serving ignition. The bootstrap node should not be showing ignition related errors. If it is, then you need to verify that the ignition server is accessible and hosting the ignition files at the intended path.
• Is the load balancer targeting the correct IP address and ports? Verify that port 6443 has the bootstrap and master nodes as targets on port 6443. Confirm the same for port 22623.

An EOF possibly sounds like the installer is able to reach the load balancer, but there are no active targets.

[xiaosayishi]

Thank you for your help. I installed it again and this problem did not appear. There may be a problem with my operation.