Preparation and Deployment of Fault-Tolerant Cluster

Clusters are used to distribute traffic, support databases, store files and business applications in the network. To provide horizontal scaling of all system components, a new approach to cluster deployment is implemented.

Advantages of new approach:

• Use new tools for deployment and service of cluster components without system downtime for maintenance.

• Use freeware with paid options.

• Support Ceph modern distributed file system.

• Quick installation and easy service of Foresight Mobile Platform.

• Friendly user interface Rancher for diagnostics of the current software state.

Deployment Architecture Schema

Main stages of fault-tolerant cluster deployment that are given in this article:

1. Prepare environment for Foresight Mobile Platform.

2. Prepare to start Foresight Mobile Platform.

3. Start Foresight Mobile Platform.

Generalized schema of fault-tolerant cluster architecture deployment:

 

The schema displays work of Kubernetes system services on main cluster nodes and also interaction of main nodes with work nodes and incoming requests via special services with load balancing support.

Schema of providing control cluster part fault-tolerance:

NOTE. Each main node in the cluster contains the following processes and components:
1. kube-apiserver. Single control point for cluster. The kubectl command interacts directly via API.
2. kube-controller-manager. Process of cluster status control via controllers.
3. kube-scheduler. Process of task planning on all available cluster nodes.
4. etcd. Database based on the key-value pairs that stores information about statuses of all cluster components.

Technical Specifications

Six virtual or physical nodes are required to create a minimum fault-tolerant cluster configuration. Each node should have at least two network interfaces.

Three nodes are main ones and are used exclusively for cluster control, cluster integrity control, planning and commissioning application containers in cluster units. Main nodes must meet the following system requirements:

• Configuration: 4-core processor, 8 GB RAM, 50 GB hard drive.

• Operating system: Linux Ubuntu 16.04.

Three nodes are work ones and execute main functions (these nodes use application units). Work nodes must meet the following system requirements:

• Configuration: 8-core processor, 8 GB RAM, two hard drives 100 GB and 150 GB each.

• Operating system: Linux Ubuntu 16.04.

Configurable software environment: Docker, Kubernetes and Ceph (if required).

The /var/lib/docker directory must be connected to a separate hard drive or LVM.

NOTE. Main nodes will be further designated as kn0, kn1, kn2, work nodes will be designated as kn3, kn4, kn5.

Preparing Environment for Foresight Mobile Platform

Preparing Nodes

To prepare nodes:

1. Create the fmpadmin user at all nodes who will be used to set and control the future cluster:

Add the fmpadmin user to sudo users to avoid entering password each time on using the sudo command:

2. Disable SWAP partition on all cluster nodes because kubelet does not support work with SWAP:

IMPORTANT. If all cluster nodes are virtual machines operated with VMware, install vmware-tools package on each node:
 sudo apt-get install -y open-vm-tools

3. Install basic packages and environment of Docker Community Edition:

To view the list of Docker Community Edition versions available for installation, execute the command:

Example of command execution result:

4. Add the fmpadmin user to the docker group:

5. Set up the file /etc/hosts identically at all nodes:

6. Generate SSH key for the fmpadmin user at the kn0 node:

1. Execute the command as the fmpadmin user:

The dialog is displayed in the console, for example:

Press the ENTER key. Then it is prompted to enter a pass phrase for additional protection of SSH connection:

Skip this step and the next step. To do this, press the ENTER key. As a result, a SSH key is created.

2. Create a configuration file for SSH:

Configuration file contents:

Save changes and exit the editor.

3. Change file permissions:

7. Collect public keys of all nodes (it is executed on the kn0 node under the fmpadmin user):

8. Add the created SSH key to all nodes:

NOTE. When password is requested, enter the password of the fmpadmin user.

9. Set up NTP server to synchronize time between the nodes:

NOTE. The setup should be executed for the second interface that is used only for inter-node traffic.

At the kn0 node:

At other nodes:

As a result, the nodes are prepared.

Installing Heartbeat Package

Target nodes for the Heartbeat package are three work nodes with a common virtual IP address. Continuous availability of services is ensured by the Heartbeat package.

NOTE. The Heartbeat package can be installed ands set up on the kn3, kn4 and kn5 nodes.

To install and set up the Heartbeat package:

1. Install and set up Heartbeat:

NOTE. All three servers share the same IP address - <main external cluster ip>. This virtual IP address is to be moved between servers that is why you should activate the net.ipv4.ip_nonlocal_bind parameter to enable binding of system services to a non-local IP address.

2. Add this option to the file /etc/sysctl.conf for the kn3, kn4, kn5 nodes:

3. Create the file /etc/ha.d/authkeys. This files stores Heartbeat files for mutual authentication, and the file must be identical on both servers:

4. Make sure that the file /etc/ha.d/authkeys is available only for the Root user:

5. Create main configuration files for Heartbeat on the selected servers (files slightly differ for each server):

1. Create the file /etc/ha.d/ha.cf on the kn3, kn4, kn5 nodes.

NOTE. To get configuration node parameters, execute uname -n on all three servers. Also use the name of your network card instead of ens160.

2. Create the file /etc/ha.d/haresources. It contains a common virtual IP address and sets which node is the main one by default. The file must be identical at all nodes:

6. Start Heartbeat services on all nodes and make sure that the specified virtual IP address is set on the kn3 node:

7. Check access by main cluster ip:

The Heartbeat package is successfully installed and set up.

Preparing Ceph System Cluster

Installing Ceph

To install and set up Ceph:

1. Install python:

NOTE. All commands must be executed on one node.

2. Execute commands:

3. Deploy Ceph first at the kn0 node:

The ceph.conf file is created in the current directory.

Edit the ceph.conf file according to the file ./fmp_k8s_v1/ceph/ceph.conf included in the distribution package:

NOTE. Replace all names and addresses of nodes with your own ones.

4. Install Ceph packages on all nodes:

This will take long time.

5. Set up monitoring of the kn1 node:

This command will create a monitoring role key. To get key, execute the command:

Adding OSD Nodes

After Ceph is installed, add OSD roles to the Ceph cluster on all nodes. They will use the /dev/sdb hard drive section to store data and log.

1. Check if the /dev/sdb hard drive section available at all nodes:

2. Delete section tables at all nodes:

This command will delete all data from the /dev/sdb section on OSD Ceph nodes.

3. Prepare and activate all OSD nodes, make sure that there are no errors:

4. Send admin keys and configuration to all nodes:

5. Change key file permissions. Execute the command at all nodes:

As a result, a Ceph cluster is created.

IMPORTANT. Add Ceph Manager Service.

To add Ceph Manager Service:

1. Execute the command on the kn0 (ceph-admin) node:

The command will return the mgr keyring value. Copy the value and go to the next step.

2. Execute the command:

Then edit the file vi /var/lib/ceph/mgr/ceph-kn1/keyring. Insert the mgr keyring value from the previous step.

3. Execute the command:

4. Start ceph-mgr manager:

5. Check status of the Ceph cluster:

If the cluster returned HEALTH_OK, the cluster works.

Creating a Metadata Server

The use of CephFS requires at least one metadata server. Execute the command to create it:

Deploying Ceph Managers

A Ceph manager works in the active or standby mode. Deploying additional Ceph managers ensures that if one of Ceph managers fails, other Ceph manager can continue operations without maintenance.

To deploy additional Ceph managers:

1. Execute the command:

Execute the following commands to check Ceph:

• Get state of Ceph cluster:

• View all existing pools of the occupied space.

• View OSD nodes status:

2. Create a single kube pool for the Kubernetes (k8s) cluster:

3. Check the number of pool replicates:

4. Check the number of pool placement groups:

5. Bind the created pool with RBD application to use it as a RADOS device:

6. Create a separate user for the kube pool and save keys that are required to allow access k8s to the storage:

7. Convert file contents to the following:

8. Convert file contents to the following:

The Ceph distributed file system is successfully installed and set up.

Preparing Kubernetes (k8s) Cluster

Execute all commands on main Kubernetes nodes under the root user. Due to possible issues with incorrect traffic routing using iptables bypass when RHEL / CentOS 7 is used, make sure that net.bridge.bridge-nf-call-iptables in the sysctl configuration is set to 1.

Execute the following at each node:

1. Make changes to the system as the root user to ensure correct work of networks in k8s:

2. Add the official Kubernetes GPG key to the system:

3. Add a Kubernetes repository:

4. Set the value /proc/sys/net/bridge/bridge-nf-call-iptables that is equal to 1 to ensure correct work of CNI (Container Network Interface). To do this, check the current value:

If the current value is 0, execute the command:

Initialize the Kubernetes cluster on the kn0 node under the root user:

1. Go to the home directory of the fmpadmin user: /home/fmpadmin. Unpack the archive with scripts and yaml files on the same controlling node:

2. Go to the directory with unpacked scripts:

3. Go to the rke subdirectory:

4. Execute the command:

The list of files in the current directory is displayed:

5. Move or copy the rke file to the /usr/local/bin/ directory:

6. Set permissions for the rke file:

7. Edit contents of the fmpclust.yml file according to your configuration:

NOTE. 10.99.255.254 — common IP address of servers cluster (if applicable); m1, m2, m3 - names of Kubernetes main nodes; w1,w2, w3 - names of Kubernetes work nodes; your user - the user, from whom interaction between nodes is made (fmpadmin).

8. Deploy and initialize the cluster:

This takes considerable time and all details of cluster deployment stages are displayed in the console.

If everything is correct, the cluster is successfully deployed, and the server console displays the string:

If there are errors, execute rke up with the debug key to display detailed information.

9. After cluster initialization, the kube_config_fmpclust.yml file appears in the current directory next to the fmpclust.yml file. Move or copy the file to the user profile. This will allow the user to interact with the cluster. Execute this operation as the fmpadmin user:

As a result, the Kubernetes (k8s) cluster is installed and initialized. Check its performance:

1. Check server and k8s client versions:

The following is displayed in the console:

2. Check status of the k8s component:

The following is displayed in the console:

Installing Rancher

Rancher is a software for easy control of k8s cluster.

To install and set up Rancher:

1. Create a folder to store Rancher information at the kn4 node:

2. Start Rancher with mounting to host:

3. Open the browser and open the URL: https://ip-address-of-kn0:8446, where ip-address-of-kn0 is IP address of the kn0 node. Set password for the Rancher administrator role in the web interface:

4. Add the created cluster to Rancher. Click the Add Cluster button:

5. Select addition type. Click the Import button:

6. Specify name and description for the added cluster and click the Create button:

7. Copy the last command on the page:

General command view:

Where <server ip> is IP address of the node.

8. Execute the copied command in the console of the kn0 node. The following is displayed in the console:

9. Wait until the system starts:

After the initialization ends, cluster status changes to Active:

General state of clusters:

10. Go to the Projects/Namespaces section and create Project for the platform:

11. Click the Add Project button. Fill in the Project Name box, add description in the Description box and click the Create button:

12. Click the Add Namespace button. Fill in the Name box and click the Create button:

The setup in the graphic web interface is completed. Go to the nodes console.

Preparing to Start Foresight Mobile Platform

1. Open the console of one of the Kubernetes main nodes as the fmpadmin user. The archive with application container images and the archive with scripts and yaml files used for starting the application in the Kubernetes environment are loaded to the user's home directory.

2. Copy the fmp_v<version number>.tgz archive to cluster work nodes.

3. Load container images to the system:

NOTE. Unpack this archive on all cluster work nodes.

After the successful import delete the archive.

4. Unpack the archive with scripts and yaml files at one control node:

5. Go to the directory with unpacked scripts:

6. Edit contents of the yaml file ./storage/ceph-storage-ceph_rbd.yml:

In the monitors string specify IP addresses or DNS names of nodes of Ceph cluster with specified monitoring role. To view which clusters have this role specified, execute the command:

The result of command execution, for example:

The "mon" string in the "services" section indicates names of the nodes with specified monitoring role.

7. Add secret-keys of distributed Ceph file storage in Kubernetes:

1. Get key of Ceph administrator:

Copy the key to the clipboard.

IMPORTANT. Avoid spaces when copying and pasting key to the command to add to Kubernetes settings!

2. Add the output key to Kubernetes secrets by explicitly specifying it in the command:

Command execution result:

3. Create a single pool for Kubernetes nodes in the Ceph cluster. The created pool will be used in RBD at nodes:

4. Create a client key. Ceph authentication is enabled in the Ceph cluster:

5. Get the client.kube key:

6. Create a new secret in the project namespace:

After adding both secrets one can start Foresight Mobile Platform.

Starting Foresight Mobile Platform

Execute scripts in the following order in the fmp_k8s_v<version number> directory:

1. create-rbds.sh. The script creates and prepares volumes for the fmp application in Ceph (it is used if there is Ceph).

2. fmp_k8s_storage_inst.sh. The script is executed if the Ceph distributed file storage is used. See above how to set up Ceph cluster.

3. Go to the volumes directory. Open the set_my_monitors.sh script for edit and replace IP addresses in it addresses of your nodes with the Ceph Monitor role in strings:

4. Execute the set_my_monitors.sh script in the volumes directory.

5. Come back to the enclosing directory.

6. fmp_k8s_volumes_inst.sh. The script creates and prepares volumes to be used in the fmp application.

7. fmp_k8s_configmap_inst.sh. The script creates a variables map.

8. fmp_k8s_services_inst.sh. The script sets up services to provide fmp application components interaction.

9. fmp_k8s_deployments_inst.sh. The script sets up conditions to start components of the fmp application.

10. Init pod. On the first startup, internal fmp services are initialized, which requires to start a separate Init pod. As soon as Init pod executes all necessary procedures, it turns to the Succeeded state.

Go back to the web interface and make sure that all required objects are created and application containers are running without errors.

1. Check if the storage is connected. Go to the Storage > Storage Classes section:

2. Check if persistent volumes are created. Go to the Storage > Persistent Volumes section:

Then select the fmp application and go to the Workbooks section to the Volumes tab:

3. Check if config maps are present that is a list of loaded variables. Go to the Resources section:

4. Go to the Workbooks section and make sure that elements are created on the tabs:

• Workloads:

• Service Discovery:

• Load Balancing is successfully turned into the Active state:

As a result, a fault-tolerant cluster and Foresight Mobile Platform.

Open the browser and follow the IP address <main cluster ip> specified on HAProxy setup. The login dialog box of Foresight Mobile Platform opens:

List of Ports Used for Cluster Work

Ports used by Kubernetes for main cluster nodes functioning are displayed in the table:

Protocol	Search direction	Range of ports	Purpose	Use
TCP	inbound	6443	Kubernetes API server	all
TCP	inbound	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	inbound	10250	kubelet API	self, control plane
TCP	inbound	10251	kube-scheduler	self
TCP	inbound	10252	kube-controller-manager	self

Ports used by Kubernetes for cluster work nodes functioning are displayed in the table:

Protocol	Search direction	Range of ports	Purpose	Use
TCP	inbound	10250	kubelet API	self, control plane
TCP	inbound	30000-32767	nodePort Services	all

RKE (Rancher Kubernetes Engine) node - Ports for outgoing connections are displayed in the table:

Protocol	Source	Range of ports	Purpose	Description
TCP	RKE node	22	all nodes specified in cluster configuration file	setup of node via SSH executed by RKE
TCP	RKE node	6443	controlling nodes	Kubernetes API server

NOTE. In this article, roles of main nodes and etcd nodes are combined.

Ports for incoming connections of controlling nodes are displayed in the table:

Protocol	Source	Range of ports	Description
TCP	any that consumes Ingress services	80	ingress controller (HTTP)
TCP	any that consumes Ingress services	443	ingress controller (HTTPS)
TCP	rancher nodes	2376	Docker daemon TLS port used by Docker Machine (only needed when using Node Driver/Templates)
TCP	etcd nodes; control plane nodes; worker nodes	6443	Kubernetes API server
UDP	etcd nodes; controlplane nodes; worker nodes	8472	canal/flannel VXLAN overlay networking
TCP	control plane node itself (local traffic, not across nodes)	9099	canal/flannel livenessProbe/readinessProbe
TCP	control plane nodes	10250	kubelet
TCP	control plane node itself (local traffic, not across nodes)	10254	ingress controller livenessProbe/readinessProbe
TCP/UDP	any source that consumes NodePort services	30000-32767	NodePort port range

Ports for incoming connections of controlling nodes are displayed in the table:

Protocol	Purpose	Range of ports	Description
TCP	rancher nodes	443	rancher agent
TCP	etcd nodes	2379	etcd client requests
TCP	etcd nodes	2380	etcd peer communication
UDP	etcd nodes; control plane nodes; worker nodes	8472	canal/flannel VXLAN overlay networking
TCP	control plane node itself (local traffic, not across nodes)	9099	canal/flannel livenessProbe/readinessProbe
TCP	etcd nodes; control plane nodes; worker nodes	10250	kubelet
TCP	control plane node itself (local traffic, not across nodes)	10254	ingress controller livenessProbe/readinessProbe

Ports for incoming connections of work nodes are displayed in the table:

Protocol	Source	Range of ports	Description
TCP	any network that you want to be able to remotely access this node from	22	remote access over SSH
TCP	any that consumes Ingress services	80	ingress controller (HTTP)
TCP	any that consumes Ingress services	443	ingress controller (HTTPS)
TCP	rancher nodes	2376	Docker daemon TLS port used by Docker Machine (only needed when using Node Driver/Templates)
UDP	etcd nodes; control plane nodes; worker nodes	8472	canal/flannel VXLAN overlay networking
TCP	worker node itself (local traffic, not across nodes)	9099	canal/flannel livenessProbe/readinessProbe
TCP	control plane nodes	10250	kubelet
TCP	worker node itself (local traffic, not across nodes)	10254	ingress controller livenessProbe/readinessProbe
TCP/UDP	any source that consumes NodePort services	30000-32767	NodePort port range

Ports for incoming connections of work nodes are displayed in the table:

Protocol	Purpose	Range of ports	Description
TCP	rancher nodes	443	rancher agent
TCP	control plane nodes	6443	Kubernetes API server
UDP	etcd nodes; control plane nodes; worker nodes	8472	canal/flannel VXLAN overlay networking
TCP	worker node itself (local traffic, not across nodes)	9099	canal/flannel livenessProbe/readinessProbe
TCP	worker node itself (local traffic, not across nodes)	10254	ingress controller livenessProbe/readinessProbe