Preparation and Deployment of Fault-Tolerant Cluster

Clusters are used to distribute traffic, support databases, store files and business applications in the network. To provide horizontal scaling of all system components, a new approach to cluster deployment is implemented.

Advantages of new approach:

• Use new tools for deployment and service of cluster components without system downtime for maintenance.

• Use freeware with paid options.

• Support SEPH modern file distributed storage system.

• Quick installation and easy service of Foresight Mobile Platform.

• Friendly user interface Rancher for diagnostics of the current software state.

Deployment Architecture Schema

Scheme of providing control cluster part fault-tolerance:

Preparing Nodes for Cluster Deployment

Creating a minimum configuration of fault-tolerant cluster requires five virtual or hardware nodes:

• Three nodes are main ones and are used exclusively for cluster control. Main nodes must meet the following system requirements:

• Configuration: 2-core processor, 8 GB RAM, 50 GB hard drive.

• Operating system: Linux Ubuntu 16.04.

• Two nodes are working ones and execute main functions. Working nodes must meet the following system requirements:

• Configuration: 4-core processor, 8 GB RAM, 500 GB hard drive.

• Operating system: Linux Ubuntu 16.04.

NOTE. Later, main nodes will be marked with the names: kn0, kn1, kn2, working nodes will be marked with the names: kn3, kn4.

To prepare nodes:

1. Create the fmpadmin user at all nodes who will be used to set and control the future cluster:

Add the fmpadmin user to sudo users to avoid entering password each time on using the sudo command:

2. Disable SWAP section at all nodes because kubeadm does not support working with SWAP:

IMPORTANT. If all cluster nodes are virtual machines controlled with VMware, install the vmware-tools package to each node:
  % sudo apt-get install -y open-vm-tools

3. Install basic packages and environment of Docker Community Edition:

To view the list of Docker Community Edition versions available for installation, execute the command:

Example of command execution result:

4. Add the fmpadmin user to the docker group:

5. Set up the file /etc/hosts identically at all nodes:

6. Generate SSH key for the fmpadmin user at the kn0 node:

1. Execute the command as the fmpadmin user:

The dialog is displayed in the console, for example:

Press the ENTER key. Then it is prompted to enter a pass phrase for additional protection of SSH connection:

Skip this step and the next step. To do this, press the ENTER key. As a result, a SSH key is created.

2. Create a configuration file for SSH:

Configuration file contents:

Save changes and exit the editor.

3. Change file permissions:

7. Add the created SSH key to all nodes:

NOTE. When password is requested, enter the password of the fmpadmin user.

8. Set up NTP server to synchronize time between the nodes:

At the kn0 node:

At other nodes:

As a result, the nodes are prepared.

Setting Up HAProxy and Heartbeat

To distribute traffic between three main nodes of Kubernetes, set up the HAProxy load balancer. As a result, three HAProxy server are obtained with a common virtual IP address. Fault-tolerance is ensured with the Heartbeat package.

NOTE. HAProxy is set up at the kn0, kn1 and kn2 main nodes.

To install and set up HAProxy using Heartbeat:

1. Install and set up HAProxy:

2. Save the source configuration and create a new one:

3. Add configuration parameters for HAProxy:

Where <main cluster ip> is a common virtual IP address that will be moved between three HAProxy servers.

4. To enable binding of system services to a non-local IP address:

1. Enable the parameter net.ipv4.ip_nonlocal_bind to the file /etc/sysctl.conf:

2. Execute the command:

3. Run HAProxy:

Make sure that HAProxy are running and listened by virtual IP address at the selected nodes:

5. Install Heartbeat and set up virtual IP address:

Execute identical installation for the kn0 and kn1 nodes.

6. Create configuration files for Heartbeat:

1. Create the file /etc/ha.d/authkeys. This file stores Heartbeat data for mutual authentication. The file must be identical at all nodes:

The created file must be available only for the root user:

2. Create the file /etc/ha.d/ha.cf at the kn1 and kn2 nodes. This file stores main Heartbeat configuration. The file will be a little different for each node.

NOTE. To get node parameters for configuration, run uname -n at each of them. Also use the name of your network card instead of ens160.

3. Create the file /etc/ha.d/haresources. It contains a common virtual IP address and sets which node is the main one by default. The file must be identical at all nodes:

7. Start Heartbeat services at all nodes and check if the specified virtual IP address is set at the kn0 node:

8. Check HAProxy performance:

Timeout must be exceeded because Kubernetes API is not yet listened at the back end. It means that HAProxy and Heartbeat are set up correctly.

Installing and Setting Up Ceph

To install and set up Ceph:

1. Install python:

2. Execute commands:

3. Deploy Ceph first at the kn0 node:

Edit the ceph.conf file:

NOTE. Replace 10.30.217.0/24 with your network address.

4. Install ceph packages at all nodes:

This will take long time.

5. Set up monitoring of the kn1 node:

This command will create a monitoring role key. To get key, execute the command:

Adding OSD Nodes in Ceph Cluster

After Ceph is installed, add OSD nodes to the cluster. They will use the /dev/sdb hard drive section to store data and log.

1. Check if the /dev/sdb hard drive section available at all nodes:

2. Delete section tables at all nodes:

This command will delete all data from the /dev/sdb section of OSD nodes.

3. Prepare and activate all OSD nodes:

Check sdb disk at OSD nodes:

4. Send admin keys and configuration to all nodes:

5. Change key file permissions. Execute the command at all nodes:

As a result, a Ceph cluster is created.

Add a Ceph Manager Service:

1. Execute the command at the kn0 node:

The command will return the mgr keyring value. Copy the value and go to the next step.

2. Execute the command:

Then edit the file vi /var/lib/ceph/mgr/ceph-kn1/keyring. Insert the mgr keyring value from the previous step.

3. Execute the command:

4. Check status of the Ceph cluster:

If the cluster returned HEALTH_OK, the cluster works.

Creating a Metadata Server

The use of CephFS requires at least one metadata server. Execute the command to create it:

Deploying Ceph Managers

A Ceph manager works in the active or standby mode. Deploying additional Ceph managers ensures that if one of Ceph managers fails, other Ceph manager can continue operations without maintenance.

To deploy additional Ceph managers:

1. Execute the command:

Execute the following commands to check Ceph:

• Get state of Ceph cluster:

• View all existing pools:

• Display CRUSH tree with hosts for OSD nodes:

2. Create a single kube pool for the Kubernetes (k8s) cluster:

3. Bind the created pool with RBD application to use it as a RADOS device:

4. Create a single user from the pool and save the keys required to ensure k8s access to the storage:

Testing Ceph Storage

The easiest way to check if a Ceph cluster works correctly and provides storage is to create and use a new RADOS Block Device (RBD) volume at the administrator node. To do this:

1. Set up RBD functions: disable the RBD functions that are not available at the used Linux kernel. To do this, edit the ceph.conf file:

IMPORTANT. Kubernetes uses RBD kernek unit to map RBD with hosts. The Ceph Luminous release requires CRUSH_TUNABLES 5 (Jewel). The minimum kernel version for these settings is 4.5. If your kernel does not support these parameters, execute the command:
ceph osd crush tunables hammer
For details see Ceph documentation, the Create and Initialize the RBD Pool section.

2. Create an RBD pool:

3. Make sure that the RBD volume exists:

4. Get information about the RBD volume:

5. Map the created RBD volume with the administrator node:

6. Create a temporary folder to connect the RBD volume, a file system of the RBD volume and connect the RBD volume to the temporary folder:

7. Check if everything works:

8. Remove the RBD volume:

Preparing to Install and Initialize Kubernetes (k8s) Cluster

Execute all commands at main Kubernetes nodes as the root user.

Execute the following at each node:

1. Make changes to the system as the root user to ensure correct work of networks in k8s:

2. Add the official Kubernetes GPG key to the system:

3. Add a Kubernetes repository:

4. Set the value /proc/sys/net/bridge/bridge-nf-call-iptables that is equal to 1 to ensure correct work of CNI (Container Network Interface). To do this, check the current value:

If it is equal to 0, execute the command:

5. The kubernetes cluster is initialized at the kn0 node:

The Kubernetes cluster is initialized at the kn0 node as the root user:

1. Go to the home directory of the fmpadmin user: /home/fmpadmin). Unpack the archive with scripts and yaml files:

2. Go to the directory with unpacked scripts:

3. Go to the rke subdirectory:

4. Execute the command:

The list of files in the current directory is displayed:

5. Move or copy the file rke, tiller, helm to the directory /usr/local/bin/:

6. Deploy and initialize the cluster:

This takes considerable time and all details of cluster deployment stages are displayed in the console.

If everything is correct, the cluster is successfully deployed, and the console displays the string:

7. After cluster initialization, the kube_config_fmpclust.yml file appears in the current directory next to the fmpclust.yml file. Move or copy the file to the user profile. This will allow the user to interact with the cluster. Execute this operation as the fmpadmin user:

As a result, the k8s cluster will be installed and initialized. Check its performance:

1. Check server and k8s client versions:

The following is displayed in the console:

2. Check status of the k8s component:

The following is displayed in the console:

Installing and Setting Up Rancher

Rancher is a software for convenient k8s cluster management.

To install and set up Rancher:

1. Create a folder to store Rancher information at the kn4 node:

2. Start Rancher with mounting to host:

3. Open the browser and open the URL: https://ip-address-of-kn0:8446, where ip-address-of-kn0 is IP address of the kn0 node. Set password for the Rancher administrator role in the web interface:

4. Add the created cluster to Rancher. Tap the Add Cluster button:

5. Select addition type. Tap the Import button:

6. Specify name and description for the added cluster and tap the Create button:

7. Copy the last command on the page:

General command view:

Where <server ip> is IP address of the node.

Execute the copied command in the console of the kn0 node. The following is displayed in the console:

Wait until the system starts:

After the initialization ends, cluster status changes to Active:

General state of clusters:

7. Go to the section Projects/Namespaces and create a project for the platform:

8. Tap the Add Project button. Fill in the Project Name box, add description in the Description box and tap the Create button:

9. Tap the Add Namespace button. Fill in the Name box and tap the Create button:

The setup in the graphic web interface is completed. Go to the nodes console.

1. Open the console of one of the Kubernetes main nodes as the fmpadmin user. The archive with application container images and the archive with scripts and yaml files used for starting the application in the Kubernetes environment are loaded to the user's home directory.

2. Load container images to the system:

NOTE. Unpack this archive at all cluster nodes.

After the successful import delete the archive.

3. Unpack the archive with scripts and yaml files at one control node:

4. Go to the directory with unpacked scripts:

5. Edit contents of the yaml file ./storage/ceph-storage-ceph_rbd.yml:

In the monitors string specify IP addresses or DNS names of nodes of ceph cluster with specified monitoring role. To view which clusters have this role specified, execute the command:

The result of command execution, for example:

The "mon" string in the "services" section indicates names of the nodes with specified monitoring role.

6. Add secret-keys of our distributed Ceph file storage in Kubernetes:

1. Get key of ceph administrator:

Copy the key to the clipboard.

2. Add the obtained key to Kubernetes secrets by explicitly specifying it in the command:

Command execution result:

3. Create a single pool for Kubernetes nodes in the Ceph cluster. The created pool will be used in RBD at nodes:

4. Create a client key. The cephx authentication is enabled in the Ceph cluster:

5. Get the client.kube key:

6. Create a new secret in our project's namespace:

7. Execute scripts in the following order:

1. fmp_k8s_storage_inst.sh. Execute when a distributed Ceph file storage is used. To set up the Ceph cluster, see above.

2. fmp_k8s_volumes_inst.sh. It creates and prepares volumes to be used by fmp application.

3. fmp_k8s_configmap_inst.sh. It creates a variables map.

4. fmp_k8s_services_inst.sh. It sets up services to enable mutual application components interaction.

5. fmp_k8s_statefulsets_inst.sh. It sets up conditions to enable database work inside fmp application.

6. fmp_k8s_deployments_inst.sh. It sets up conditions to start components of fmp application.

Go back to the web interface and make sure that all required objects are created and application containers are running without errors.

1. Check if the storage is connected. Go to the Storage > Storage Classes section:

2. Check if persistent volumes are created. Go to the Storage > Persistent Volumes section:

Then select fmp application and go to the Workbooks section to the Volumes tab:

3. Check if config maps are present that is a list of loaded variables. Go to the Resources section:

4. Go to the Workbooks section and make sure that elements are created on the tabs:

• Workloads:

• Service Discovery:

• Load Balancing is successfully turned into the Active state:

As a result, a fault-tolerant cluster and Foresight Mobile Platform application are prepared and deployed.

Open the browser and follow the IP address <main cluster ip> specified on HAProxy setup. The login dialog box of Foresight Mobile Platform opens: