To deploy a fault-tolerant cluster based OKD/OCP, prepare and environment and deploy auxiliary nodes:
Create a single VLAN OKD-prod in vCentre (WUI of virtualization environment) for cluster network.
Load and install the VMware template to the virtualization environment.
Reserve IP addresses for the srv and rt hosts if DHCP server is connected in the Office LAN network, or use static IP addressing.
Create the okd.lan zone on the DNS server in the Office LAN network and the prod domain contained in this zone. After executing the operation add DNS records:
Type | Name | Purpose |
A | srv.prod.okd.lan | srv Host IP address |
A | rt.prod.okd.lan | rt Host IP address |
CNAME | * | srv.prod.okd.lan |
NOTE. Domain names are specified for the example of environment preparation and deployment of auxiliary nodes and may differ.
Deploy auxiliary nodes:
After executing the operations, the environment is prepared and auxiliary nodes are deployed for the cluster. Proceed to cluster deployment.
To deploy router:
Create a virtual machine according to the system requirements and set up switching of virtual machine network interfaces according to the cluster components interaction scheme.
Load pfSense (architecture/AMD64) to iso disk and connect it to the virtual machine. For details about the use of pfSense see the documentation.
Start the virtual machine and install pfSense loaded from the iso disk. Use default values on installation.
Restart the virtual machine and disconnect the iso disk.
Open pFsense Console (Shell) and execute the following operations one by one:
Create an allow rule to access the router from the Office LAN network using the command:
easyrule pass wan tcp any any 443
Disconnect packet filter using the command:
pfctl –d
Connect to the WUI web interface at:
https://rt.prod.okd.lan
Execute the following operations in the WUI web interface one by one:
Set host name and set up network interfaces.
Disable blocking of IP addresses of private networks in the WAN interface using the command: Interfaces > WAN (vmx0) > Block RFC1918 Private Networks.
Install Open-VM-Tools using the command: System > Package Manager > Available Package > Open-VM-Tools.
Install updates using the command: System > Update > System Update.
After executing the operations the router is deployed.
To deploy service node:
Create a virtual machine according to the system requirements and set up switching of virtual machine network interfaces according to the cluster components interaction scheme.
Load Fedora Server Standard (33/Server/x86_64) to iso disk and connect it to the virtual machine. For details about the use of Fedora Server Standard see the documentation.
Start the virtual machine and install the loaded Fedora Server Standard from the iso disk. Use default values on installation.
Restart the virtual machine and disconnect the iso disk.
Connect to the WUI web interface at:
https://srv.prod.okd.lan:9090
Execute the following operations in the WUI web interface one by one:
Set host name and set up network interfaces.
Install updates using the command: System > Update > System Update.
Install the following applications:
DNS server. Execute the commands:
sudo dnf -y install bind bind-utils
sudo cat <<EOF > /etc/named.conf
options {
listen-on port 53 { any; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
secroots-file "/var/named/data/named.secroots";
recursing-file "/var/named/data/named.recursing";
allow-query { any; };
recursion yes;
forwarders {
10.9.0.5;
10.9.0.6;
};
dnssec-enable yes;
dnssec-validation no;
managed-keys-directory "/var/named/dynamic";
geoip-directory "/usr/share/GeoIP";
pid-file "/run/named/named.pid";
session-keyfile "/run/named/session.key";
include "/etc/crypto-policies/back-ends/bind.config";
};
logging {
channel default_debug {
file "data/named.run";
severity dynamic;
};
};
zone "." IN {
type hint;
file "named.ca";
};
zone "prod.okd.lan" IN {
type master;
file "zonefile.db";
};
zone "0.168.192.in-addr.arpa" IN {
type master;
file "reverse.db";
};
include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
EOF
sudo cat <<EOF > /var/named/zonefile.db
$TTL 86400
@ IN SOA ns1.prod.okd.lan. root (
1614163446 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS ns1.prod.okd.lan.
ns1 IN A 192.168.0.2
api IN A 192.168.0.2
api-int IN A 192.168.0.2
*.apps IN A 192.168.0.2
srv IN A 192.168.0.2
bs IN A 192.168.0.10
mst-1 IN A 192.168.0.11
mst-2 IN A 192.168.0.12
mst-3 IN A 192.168.0.13
wrk-1 IN A 192.168.0.14
wrk-2 IN A 192.168.0.15
etcd-0 IN A 192.168.0.11
etcd-1 IN A 192.168.0.12
etcd-2 IN A 192.168.0.13
_etcd-server-ssl._tcp IN SRV 0 10 2380 etcd-0.prod.okd.lan.
_etcd-server-ssl._tcp IN SRV 0 10 2380 etcd-1.prod.okd.lan.
_etcd-server-ssl._tcp IN SRV 0 10 2380 etcd-2.prod.okd.lan.
EOF
sudo cat <<EOF > /var/named/reverse.db
$TTL 86400
@ IN SOA ns1.prod.okd.lan. root (
1614163446 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS ns1.prod.okd.lan.
2 IN PTR api.prod.okd.lan.
2 IN PTR api-int.prod.okd.lan.
2 IN PTR srv.prod.okd.lan.
10 IN PTR bs.prod.okd.lan.
11 IN PTR mst-1.prod.okd.lan.
12 IN PTR mst-2.prod.okd.lan.
13 IN PTR mst-3.prod.okd.lan.
14 IN PTR wrk-1.prod.okd.lan.
15 IN PTR wrk-2.prod.okd.lan.
EOF
sudo systemctl enable named; sudo systemctl start named
sudo firewall-cmd --permanent --add-port=53/udp
sudo firewall-cmd --reload
Load balancer. Execute the commands:
sudo dnf -y install haproxy
sudo cat <<EOF > /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 20000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
ssl-default-bind-ciphers PROFILE=SYSTEM
ssl-default-server-ciphers PROFILE=SYSTEM
defaults
mode tcp
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 20000
listen stats
bind :9000
mode http
stats enable
stats uri /
monitor-uri /healthz
frontend openshift-api-server
bind *:6443
default_backend openshift-api-server
option tcplog
backend openshift-api-server
balance source
server bs 192.168.0.10:6443 check
server mst-1 192.168.0.11:6443 check
server mst-2 192.168.0.12:6443 check
server mst-3 192.168.0.13:6443 check
frontend machine-config-server
bind *:22623
default_backend machine-config-server
option tcplog
backend machine-config-server
balance source
server bs 192.168.0.10:22623 check
server mst-1 192.168.0.11:22623 check
server mst-2 192.168.0.12:22623 check
server mst-3 192.168.0.13:22623 check
frontend ingress-http
bind *:80
default_backend ingress-http
option tcplog
backend ingress-http
balance source
server wrk-1-http 192.168.0.14:80 check
server wrk-2-http 192.168.0.15:80 check
frontend ingress-https
bind *:443
default_backend ingress-https
option tcplog
backend ingress-https
balance source
server wrk-1-https 192.168.0.14:443 check
server wrk-2-https 192.168.0.15:443 check
EOF
sudo setsebool -P haproxy_connect_any 1
sudo systemctl enable haproxy; sudo systemctl start haproxy
sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=22623/tcp
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload
WEB server. Execute the commands:
sudo dnf install -y httpd
sudo sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf
sudo setsebool -P httpd_read_user_content 1
sudo systemctl enable httpd; sudo systemctl start httpd
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload
After executing the operations, the service node is deployed.
See also:
Preparation and Deployment of Fault-Tolerant Cluster Based on OKD/OCP | Deployment of Cluster Based on OKD/OCP