Deployment of Cluster Based on OKD/OCP

To deploy a fault-tolerant cluster based on OKD/OCP, execute the following operations on the srv host:

  1. Install stable versions 4.6 oc client and openshift-installer (OKD) using the commands executed in the WUI web interface or in the console via the SSH connection:

VERSION=4.6.0-0.okd-2021-02-14-205305
mkdir -p ~/openshift/install 
cd ~/openshift
wget https://github.com/openshift/okd/releases/download/$VERSION/openshift-client-linux-$VERSION.tar.gz
wget https://github.com/openshift/okd/releases/download/$VERSION/openshift-install-linux-$VERSION.tar.gz
tar -zxvf  openshift-client-linux-$VERSION.tar.gz
tar -zxvf openshift-install-linux-$VERSION.tar.gz
mv -f kubectl oc openshift-install /usr/local/bin/

  1. Generate SSH keys using the command:

ssh-keygen -t ed25519 -N ''

  1. Create manifests using the commands:

cd /root/openshift/install
cat << EOF > ./install-config.yaml 
apiVersion: v1
baseDomain: okd.lan
metadata: 
name: prod
compute:
  - hyperthreading: Enabled 
name: worker 
replicas: 0
controlPlane: 
hyperthreading: Enabled 
name: master 
replicas: 3
networking: 
clusterNetwork: 
  - cidr: 10.128.0.0/14    
hostPrefix: 23  
machineNetwork: 
  - cidr: 192.168.0.0/24 
networkType: OVNKubernetes 
serviceNetwork:  
- 172.30.0.0/16
platform: 
none: {}
fips: false
pullSecret: '{"auths":{"fake":{"auth": "bar"}}}' 
sshKey: '<public SSH key>'

EOF
openshift-install create manifests
rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml openshift/99_openshift-cluster-api_worker-machineset-*.yaml
sed -i 's/mastersSchedulable: true/mastersSchedulable: false/' manifests/cluster-scheduler-02-config.yml
openshift-install create ignition-configs
cat << EOF > ./append-bootstrap.ign 
{
"ignition": {   
"config": {   
"merge": [
        {
        "source": "http://192.168.0.2:8080/openshift/bootstrap.ign" 
        }   
]   
},   
"version": "3.1.0"
 }
}
EOF
cat << EOF > ./append-master.ign 
{
"ignition": {   
"config": {   
"merge": [
        {
        "source": "http://192.168.0.2:8080/openshift/master.ign" 
        }   
]   
},   
"version": "3.1.0"
 }
}
EOF
cat << EOF > ./append-worker.ign 
{
"ignition": {   
"config": {   
"merge": [
        {
        "source": "http://192.168.0.2:8080/openshift/worker.ign" 
        }   
]   
},   
"version": "3.1.0"
 }
}
EOF
base64 -w0 append-bootstrap.ign > append-bootstrap.64
base64 -w0 append-master.ign > append-master.64
base64 -w0 append-worker.ign > append-worker.64
mkdir /var/www/html/openshift
chown -R apache:apache /var/www/html/openshift
cp ./*.ign /var/www/html/openshift/
chown -R apache:apache /var/www/html/openshift/*
chmod 644 /var/www/html/openshift/*

In the <public SSH key> substitution set the generated SSH key contained in the *.pub file.

  1. Create cluster nodes using vCenter (virtualization environment WUI):

    1. Select the VMware installed template and clone the virtual machine using the command: Clone > Clone to Virtual Machine.

    2. Specify virtual machine name according to the system requirements.

    3. Select the Customize This Virtual Machine's Hardware checkbox.

    4. Specify the required number of CPU and RAM, connect the disks according to the system requirements.

    5. Set up switching of virtual machine network interfaces according to the cluster component interaction scheme.

    6. Edit additional options of the virtual machine using the command: VM Options > Advanced > Edit Configuration. Specify the corresponding value for each virtual machine:

NOTE. Names and IP addresses of the hosts are specified for the example of deployment of cluster based on OKD/OCP and may differ.

NOTE. Values for specific node types are contained on the srv host after manifest creation.

    1. Create a virtual machine.

    2. Repeat the above mentioned operations for each virtual machine.

  1. Turn on the virtual machines in the following order:

For details about virtual machines see the OKD documentation.

  1. Execute the following commands to monitor cluster installation on the srv host:

export KUBECONFIG=~/openshift/install/auth/kubeconfig
cd ~/openshift/install
openshift-install wait-for bootstrap-complete --log-level=debug
openshift-install wait-for install-complete --log-level=debug

IMPORTANT. At the same time check issuing of certificates.

The example of issuing certificates:

oc get csr
NAME        AGE     SIGNERNAME                                    REQUESTOR                                                                   CONDITION
csr-25z84   91m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-7jpl5   22m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-7r7z2   81m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-hwjbp   37m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-lxgq5   91m     kubernetes.io/kubelet-serving                 system:node:mst-1.prod.okd.lan                                              Approved,Issued
csr-r4jgl   81m     kubernetes.io/kubelet-serving                 system:node:mst-3.prod.okd.lan                                              Approved,Issued
csr-vvdbc   81m     kubernetes.io/kubelet-serving                 system:node:mst-2.prod.okd.lan                                              Approved,Issued
csr-wj5qn   7m18s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-wlxwh   81m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-xzmms   16m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-z99h9   31m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-zql4k   89s     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending

If there are pending requests with the Pending state, confirm them using the command:

oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

  1. Make sure that all cluster nodes are in the Ready state:

oc get nodes
NAME                 STATUS   ROLES    AGE   VERSION
mst-1.prod.okd.lan   Ready    master   45m   v1.19.2+f173eb4-1049
mst-2.prod.okd.lan   Ready    master   45m   v1.19.2+f173eb4-1049
mst-3.prod.okd.lan   Ready    master   45m   v1.19.2+f173eb4-1049
wrk-1.prod.okd.lan   Ready    worker   20m   v1.19.2+f173eb4-1049
wrk-2.prod.okd.lan   Ready    worker   26s   v1.19.2+f173eb4-1049

  1. Make sure that all cluster-operators are running and available:

[root@srv install]# oc get co
NAME                                                                   VERSION                         AVAILABLE   PROGRESSING   DEGRADED SINCE
authentication                                                         4.6.0-0.okd-2021-02-14-205305   True        False         False      10s
cloud-credential                                                       4.6.0-0.okd-2021-02-14-205305   True        False         False      11h
cluster-autoscaler                                                     4.6.0-0.okd-2021-02-14-205305   True        False         False      11h
config-operator                                                        4.6.0-0.okd-2021-02-14-205305   True        False         False      11h
console                                                                4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
csi-snapshot-controller                                                4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
dns                                                                    4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
etcd                                                                   4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
image-registry                                                         4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
ingress                                                                4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
insights                                                               4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
kube-apiserver                                                         4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
kube-controller-manager                                                4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
kube-scheduler                                                         4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
kube-storage-version-migrator                                          4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
machine-api                                                            4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
machine-approver                                                       4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
machine-config                                                         4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
marketplace                                                            4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
monitoring                                                             4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
network                                                                4.6.0-0.okd-2021-02-14-205305   True        False         False      11h
node-tuning                                                            4.6.0-0.okd-2021-02-14-205305   True        False         False      11h
openshift-apiserver                                                    4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
openshift-controller-manager                                           4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
openshift-samples                                                      4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
operator-lifecycle-manager                                             4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
operator-lifecycle-manager-catalog                                     4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
operator-lifecycle-manager-packageserver                               4.6.0-0.okd-2021-02-14-205305   True        False         False      10h
service-ca                                                             4.6.0-0.okd-2021-02-14-205305   True        False         False      11h
storage                                                                4.6.0-0.okd-2021-02-14-205305   True        False         False      11h

After executing the operations the fault-tolerant cluster based on OKD/OCP is deployed. Proceed to cluster setup.

The example of output after cluster installation:

openshift-install wait-for install-complete --log-level=debug
DEBUG OpenShift Installer 4.6.0-0.okd-2021-02-14-205305
DEBUG Built from commit 20585a2903a61784703fde00e38af1208c007599
DEBUG Loading Install Config...                  
DEBUG   Loading SSH Key...                        
DEBUG   Loading Base Domain...                    
DEBUG     Loading Platform...                     
DEBUG   Loading Cluster Name...                   
DEBUG     Loading Base Domain...                  
DEBUG     Loading Platform...                     
DEBUG   Loading Pull Secret...                    
DEBUG   Loading Platform...                       
DEBUG Using Install Config loaded from state file 
INFO Waiting up to 40m0s for the cluster at https://api.prod.okd.lan:6443 to initialize...
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 0% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 7% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 8% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 63% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 64% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 66% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 86% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 86% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 86% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 86% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 86% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: downloading update
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 1% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 9% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 57% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 67% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 69% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 71% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 87% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 88% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 88% complete, waiting on authentication, console, ingress, kube-storage-version-migrator, monitoring
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 89% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.okd-2021-02-14-205305: 89% complete
DEBUG Cluster is initialized                      
INFO Waiting up to 10m0s for the openshift-console route to be created...
DEBUG Route found in openshift-console namespace: console
DEBUG Route found in openshift-console namespace: downloads
DEBUG OpenShift console route is created          
INFO Install complete!                           
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/openshift/install/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.prod.okd.lan
INFO Login to the console with user: "kubeadmin", and password: "PiXuD-zAUIt-3NTWU-JfIYo"
DEBUG Time elapsed per stage:                     
DEBUG Cluster Operators: 35m16s                   
INFO Time elapsed: 35m16s

See also:

Preparation and Deployment of Fault-Tolerant Cluster Based on OKD/OCP | Setting Up Cluster